https://prefect.io logo
j

jpuris

07/24/2023, 2:17 PM
Heya! My flow runs on a Local Process infrastructure (no block used) with a flow that runs a dbt dbt operation with
prefect-dbt
. Before the dbt run, I need to export certain env variables e.g. database password and some non-sensitive one. For the non-sensitive env vars, I can use a Process block (type process), but how would I go about using a secret block within it as an env var? Alternatively, I can bake the Secret block retrieval and hardcode the other env vars in the flow and use
os.environ
to export the env vars for the process, which is not preferable.
1
f

Farhood Etaati

07/24/2023, 2:22 PM
You can get the sensitive information from a hashicorp block. It's a great way to handle secrets.
j

jpuris

07/24/2023, 2:24 PM
I may have described my problem poorly.. It is not that I'm looking for where to put my secret. I am, however, looking for how to inject it into Environmental variables of an infrastructure block.
f

Farhood Etaati

07/24/2023, 2:25 PM
you can inject the name of the secret block into the run_deployment keys (via environment variable injection) in the code from what I recall. And then load it securely in your flow via loading the block.
infra_overrides
was the key I think.
j

jpuris

07/24/2023, 2:25 PM
That may work! Do you happen to have an example of this? 😕
when you're creating the deployment object you can declare an
infra_overrides
key in which you can nest an
env
key for dynamic environment variables.
j

jpuris

07/24/2023, 2:32 PM
I have a secret block
my-db-pass
infrastructure block
my-ec2-process
I do not want to store the value of the secret in the infrastructure block, but instead have it retrieve the env var's value from secret block instead. In the example I see a literal value provided as the
infra_overrides
Copy code
infra_overrides:
  env.EXTRA_PIP_PACKAGES: s3fs
🤷
Instead I'm looking for
Copy code
infra_overrides:
  env.EXTRA_PIP_PACKAGES: my-db-pass-secret-block-ref
Is this at all possible or do I need to load the secret from the flow and have python expose it in runtime? 😞
n

Nate

07/24/2023, 2:54 PM
just fyi, something like this would be supported by the templating options available in the
prefect.yaml
I'm not certain where you're currently trying to do this
Copy code
infra_overrides:
  env.EXTRA_PIP_PACKAGES: my-db-pass-secret-block-ref
j

jpuris

07/24/2023, 3:05 PM
Hi Nate! I am looking to expose a secret block's value as an environment variable for a Process Infrastrucure supported flow. I have not tried to do anything with
infra_overrides
as I believe it does not do what I'm looking for. My use case: 1. A secret block 2. A flow I need the secret blocks's value exposed as environment variable for the flows execution. The infrastructure is of process type (prefect agent running on an EC2). Looking at Prefect Cloud's "Infrastructure" block, I see there is an input for "Environment", but I can not reference any secret block as the environmental variables value and hence the question on how would I go about doing this.
Looking at the templating docs you've linked, I believe I can do this as one of the examples show
Copy code
"{{ prefect.blocks.aws-credentials.dev-credentials }}"
I have not tried Prefect projects yet, hence this flew under my radar. Is it safe to assume the classic deployment does not support templating?
n

Nate

07/24/2023, 3:10 PM
easiest way would be to define the deployment in a
prefect.yaml
like
Copy code
deployments:
- name: test
  entrypoint: src/prefect-case-studies/test.py:test
  work_pool:
    name: kubernetes-prd-internal-tools
    job_variables:
      env:
        MY_ENV_VAR: "{{ prefect.blocks.secret.my-secret }}"
except with a process work pool
and yeah, the new declarative way supports templating
j

jpuris

07/24/2023, 3:11 PM
This is exactly what I was looking for then. Now the only thing left is to move to using projects 🙂👍 Thank you!!
n

Nate

07/24/2023, 3:14 PM
cool - let me know if you generate any questions in moving over!
2 Views