Bharat Khaneja
08/20/2025, 9:13 PMMarvin
08/20/2025, 9:13 PMMarvin
08/20/2025, 9:19 PMenv
from /api/work_pools/filter
. That endpoint returns the full work pool model, including the base_job_template
and its job_configuration.env
if you’ve put values there. If sensitive values are stored in the pool template, they will be returned by the API.
Recommended approaches to avoid exposing secrets:
- Don’t store secrets in the work pool’s base_job_template.env
- Instead, inject secrets at runtime via your infrastructure:
- Kubernetes: reference Kubernetes Secrets (e.g., envFrom/secretKeyRef) so only the secret names appear in the template, not values.
- Docker/ECS/etc.: use the platform’s secret mechanism or pass env vars to the runtime container from your orchestrator; avoid putting the actual values in Prefect’s pool config.
- Use Prefect Secret blocks in your flow code
- Store the secret in a Prefect Secret block (encrypted at rest) and load it in your flow; don’t propagate it through the pool template.
- Example:
from prefect.blocks.system import Secret
# One-time setup
Secret(value="super-secret").save("my-api-token", overwrite=True)
# In your flow
@flow
def run():
token = Secret.load("my-api-token").get()
...
- Docs: Store secrets
- Use templating to reference env at render time (keeps values out of the database)
- In your pool’s base_job_template
, set env values to placeholders that read from the worker’s environment at runtime, e.g.:
{
"job_configuration": {
"env": {
"MY_SECRET": "{{ env.MY_SECRET }}"
}
}
}
- This way, the API returns the placeholder string, not the secret value. Ensure the worker’s environment actually has MY_SECRET
set.
- Worker/template docs: Manage work pools and Custom worker
A few additional notes:
- Prefect Variables are not intended for secrets (no sensitive/redaction flag). Avoid putting secrets there: Variables
- Work pools API reference (to confirm the returned fields): Work pools API
If you can share which worker type you’re using (Kubernetes, Docker, etc.) and whether this is Prefect Cloud or a self-hosted server, I can suggest a concrete pattern and example template for that runtime.Bharat Khaneja
08/21/2025, 4:38 AMMarvin
08/21/2025, 4:38 AMMarvin
08/21/2025, 4:46 AMprefect deploy ./path/to/script.py:flow_func \
--name my-deployment \
--job-variable env.MY_SECRET=super-secret-value
- Note: this stores the value with the deployment in Prefect’s API and may be visible to users with access. Also consider your shell history. Use only if that’s acceptable.
- Docs: Customize job variables
- Define env vars in prefect.yaml (checked into your repo)
- Add them under job_variables.env for the deployment:
deployments:
- name: my-deployment
entrypoint: path/to/script.py:flow_func
work_pool:
name: process-pool
job_variables:
env:
MY_SECRET: "super-secret-value"
- Docs: prefect.yaml reference
- Set env vars at the work pool or worker level (recommended for sensitive values)
- Process work pool: set env vars on the machine/service that runs prefect worker start
— flow runs inherit them.
- Docker/Kubernetes work pools: add env to the base job template so every run gets them (or use your orchestrator’s secret store, e.g., Kubernetes Secrets).
- Docs: Customize job variables (work pool base job template)
If you were open to a tiny code change later, Prefect provides Variables and a Secret block you can load at runtime (keeps secrets out of deployments), but that does require reading them in code:
- Variables/Secret concepts: Variables concept
Questions to tailor the best option:
- How does your script currently get secrets (env vars, config file, CLI args)?
- Which work pool/worker type are you using (process, docker, kubernetes)?
- Any constraints about storing secrets in Prefect vs only on the worker/infrastructure?