<@ULVA73B9P> I would like to have env variable def...
# ask-marvin
s
@Marvin I would like to have env variable definitions in my deployment definition in Python for a deployment going to a docker work pool. If I do
Copy code
job_variables={
                "volumes": [
                    f"{my_volume}:{my_volume}",
                ],
                "env": {
                    "MY_ENV_VAR": "{{ $MY_ENV_VAR}}",
                },
                "mem_limit": "2g",
                "auto_remove": True,
            },
Everything gets set properly except for the env var. This happens even when not templated and a constant string. As a result, I am defining this in my work pool definition as {"MY_ENV_VAR": "{{ $MY_ENV_VAR}}"}. However, I would like to have everything explicitly defined in Python in one place. Am I making a syntax error in my job_variables argument of my deployment?
m
thought for 669.5 seconds
Short answer: your syntax looks right for a Docker work pool;
env
should be a dict of strings and the other keys you’re using are valid. So you’re not making an obvious syntax error. A few specifics and checks: - For Docker work pools in Prefect 3.x, the job configuration includes: -
env: dict[str, Optional[str]]
-
volumes: list[str]
-
mem_limit: Optional[str]
-
auto_remove: bool
-
container_create_kwargs: Optional[dict[str, Any]]
- Reference: Customize job variables and Developing a custom worker - The
env
key is correct for Docker. For example:
Copy code
job_variables={
    "volumes": [f"{my_volume}:{my_volume}"],
    "env": {"MY_ENV_VAR": "hello"},
    "mem_limit": "2g",
    "auto_remove": True,
  }
This matches the Docker worker job config signature:
Copy code
DockerWorkerJobConfiguration(
    ...,
    env: dict[str, Optional[str]] = {},
    volumes: list[str] = [],
    mem_limit: Optional[str] = None,
    auto_remove: bool = False,
    container_create_kwargs: Optional[dict[str, Any]] = None,
    ...
  )
- Important templating note:
{{ $MY_ENV_VAR }}
expansion is handled when you use prefect.yaml/CLI templating. If you pass that literal string from Python, it will not be rendered; it will be passed as the literal value
{{ $MY_ENV_VAR }}
. Since you said even a constant string doesn’t show up, this isn’t the core issue—but keep this in mind if you need dynamic values from your shell at deploy time. In Python, if you want to forward an environment value, do e.g.
os.environ.get("MY_ENV_VAR")
when building
job_variables
. What to check next 1. Confirm where you’re expecting the variable to appear: -
env
is applied to the flow’s container, not to the worker process. Inside your flow, add a quick check:
Copy code
import os
     from prefect import flow

     @flow
     def debug_env():
         print("MY_ENV_VAR inside container:", os.environ.get("MY_ENV_VAR"))
2. Inspect the flow run’s job configuration in the UI (Flow run > Configuration tab). Do you see your
env
dict present there? If it’s missing there, the issue is during deployment definition. If it’s present but not visible in the container, the issue is at run/worker time. 3. Versions matter. Please share: -
prefect
version and
prefect-docker
version on your deployment machine - The worker image/version used by your Docker work pool - How you’re creating the deployment in Python (a small snippet around
.deploy(...)
) Workarounds and additional tips - As a workaround, you can also set environment variables via the Docker SDK passthrough using `container_create_kwargs`:
Copy code
job_variables={
    "env": {"MY_ENV_VAR": "hello"},  # primary, should work
    "container_create_kwargs": {
      "environment": {"MY_ENV_VAR_2": "world"}
    }
  }
The worker will forward
container_create_kwargs
to
docker.containers.create(environment=...)
. - If you have defaults on the work pool and env also set on the deployment, they should merge (deployment values take precedence). If you’re seeing “works only when defined on the work pool, ignored on the deployment”, that suggests a version-specific bug or mismatch. Sharing versions will help us reproduce. If you can paste the small deployment snippet showing how you call
deploy(...)
, plus your versions, I’ll try to spot anything off and reproduce. Here are the relevant docs for reference: - Customize job variables: Customize job variables - Docker work pools overview: Docker work pools - Custom worker/job config details: Developing a custom worker