Steve Friedman
11/11/2025, 11:08 PMjob_variables={
"volumes": [
f"{my_volume}:{my_volume}",
],
"env": {
"MY_ENV_VAR": "{{ $MY_ENV_VAR}}",
},
"mem_limit": "2g",
"auto_remove": True,
},
Everything gets set properly except for the env var. This happens even when not templated and a constant string. As a result, I am defining this in my work pool definition as {"MY_ENV_VAR": "{{ $MY_ENV_VAR}}"}. However, I would like to have everything explicitly defined in Python in one place. Am I making a syntax error in my job_variables argument of my deployment?Marvin
11/11/2025, 11:08 PMMarvin
11/11/2025, 11:20 PMenv should be a dict of strings and the other keys you’re using are valid. So you’re not making an obvious syntax error.
A few specifics and checks:
- For Docker work pools in Prefect 3.x, the job configuration includes:
- env: dict[str, Optional[str]]
- volumes: list[str]
- mem_limit: Optional[str]
- auto_remove: bool
- container_create_kwargs: Optional[dict[str, Any]]
- Reference: Customize job variables and Developing a custom worker
- The env key is correct for Docker. For example:
job_variables={
"volumes": [f"{my_volume}:{my_volume}"],
"env": {"MY_ENV_VAR": "hello"},
"mem_limit": "2g",
"auto_remove": True,
}
This matches the Docker worker job config signature:
DockerWorkerJobConfiguration(
...,
env: dict[str, Optional[str]] = {},
volumes: list[str] = [],
mem_limit: Optional[str] = None,
auto_remove: bool = False,
container_create_kwargs: Optional[dict[str, Any]] = None,
...
)
- Important templating note: {{ $MY_ENV_VAR }} expansion is handled when you use prefect.yaml/CLI templating. If you pass that literal string from Python, it will not be rendered; it will be passed as the literal value {{ $MY_ENV_VAR }}. Since you said even a constant string doesn’t show up, this isn’t the core issue—but keep this in mind if you need dynamic values from your shell at deploy time. In Python, if you want to forward an environment value, do e.g. os.environ.get("MY_ENV_VAR") when building job_variables.
What to check next
1. Confirm where you’re expecting the variable to appear:
- env is applied to the flow’s container, not to the worker process. Inside your flow, add a quick check:
import os
from prefect import flow
@flow
def debug_env():
print("MY_ENV_VAR inside container:", os.environ.get("MY_ENV_VAR"))
2. Inspect the flow run’s job configuration in the UI (Flow run > Configuration tab). Do you see your env dict present there? If it’s missing there, the issue is during deployment definition. If it’s present but not visible in the container, the issue is at run/worker time.
3. Versions matter. Please share:
- prefect version and prefect-docker version on your deployment machine
- The worker image/version used by your Docker work pool
- How you’re creating the deployment in Python (a small snippet around .deploy(...))
Workarounds and additional tips
- As a workaround, you can also set environment variables via the Docker SDK passthrough using `container_create_kwargs`:
job_variables={
"env": {"MY_ENV_VAR": "hello"}, # primary, should work
"container_create_kwargs": {
"environment": {"MY_ENV_VAR_2": "world"}
}
}
The worker will forward container_create_kwargs to docker.containers.create(environment=...).
- If you have defaults on the work pool and env also set on the deployment, they should merge (deployment values take precedence). If you’re seeing “works only when defined on the work pool, ignored on the deployment”, that suggests a version-specific bug or mismatch. Sharing versions will help us reproduce.
If you can paste the small deployment snippet showing how you call deploy(...), plus your versions, I’ll try to spot anything off and reproduce. Here are the relevant docs for reference:
- Customize job variables: Customize job variables
- Docker work pools overview: Docker work pools
- Custom worker/job config details: Developing a custom worker