jpuris
07/24/2023, 2:17 PMprefect-dbt
. Before the dbt run, I need to export certain env variables e.g. database password and some non-sensitive one.
For the non-sensitive env vars, I can use a Process block (type process), but how would I go about using a secret block within it as an env var?
Alternatively, I can bake the Secret block retrieval and hardcode the other env vars in the flow and use os.environ
to export the env vars for the process, which is not preferable.Farhood Etaati
07/24/2023, 2:22 PMjpuris
07/24/2023, 2:24 PMFarhood Etaati
07/24/2023, 2:25 PMinfra_overrides
was the key I think.jpuris
07/24/2023, 2:25 PMFarhood Etaati
07/24/2023, 2:27 PMinfra_overrides
key in which you can nest an env
key for dynamic environment variables.jpuris
07/24/2023, 2:32 PMmy-db-pass
infrastructure block my-ec2-process
I do not want to store the value of the secret in the infrastructure block, but instead have it retrieve the env var's value from secret block instead.
In the example I see a literal value provided as the infra_overrides
infra_overrides:
env.EXTRA_PIP_PACKAGES: s3fs
🤷infra_overrides:
env.EXTRA_PIP_PACKAGES: my-db-pass-secret-block-ref
Nate
07/24/2023, 2:54 PMprefect.yaml
infra_overrides:
env.EXTRA_PIP_PACKAGES: my-db-pass-secret-block-ref
jpuris
07/24/2023, 3:05 PMinfra_overrides
as I believe it does not do what I'm looking for.
My use case:
1. A secret block
2. A flow
I need the secret blocks's value exposed as environment variable for the flows execution.
The infrastructure is of process type (prefect agent running on an EC2).
Looking at Prefect Cloud's "Infrastructure" block, I see there is an input for "Environment", but I can not reference any secret block as the environmental variables value and hence the question on how would I go about doing this."{{ prefect.blocks.aws-credentials.dev-credentials }}"
I have not tried Prefect projects yet, hence this flew under my radar.
Is it safe to assume the classic deployment does not support templating?Nate
07/24/2023, 3:10 PMprefect.yaml
like
deployments:
- name: test
entrypoint: src/prefect-case-studies/test.py:test
work_pool:
name: kubernetes-prd-internal-tools
job_variables:
env:
MY_ENV_VAR: "{{ prefect.blocks.secret.my-secret }}"
jpuris
07/24/2023, 3:11 PMNate
07/24/2023, 3:14 PM