Olve Heitmann
09/13/2025, 9:54 PMMarvin
09/13/2025, 9:54 PMMarvin
09/13/2025, 10:05 PMjob_variables if you want to override or add different values for that deployment.
- Worker process environment variables are not automatically forwarded into the job container (except the Prefect-required ones like API URL/keys, etc.). If you want user-defined env vars inside the flow-run container, set them explicitly via the work pool base job template or deployment job_variables.
- You can configure pool-level defaults via the UI, CLI, Python SDK, or by scripting (e.g., in Docker Compose) — all options below.
Details and precedence
- Where to set:
- Work pool base job template: global defaults for all runs launched from that pool (recommended for mounts/env needed everywhere).
- Deployment `job_variables`: per-deployment overrides or additions.
- Precedence and merge:
- Deployment job_variables override pool defaults.
- For dict-like fields such as env, values are merged with deployment keys overriding duplicate keys.
- For list-like fields such as volumes, treat the deployment-provided list as a replacement of the pool list. If you want “pool defaults + extra” volumes, include the entire final list at the deployment or keep all mounts at the pool level.
- Automatic env propagation:
- User-defined env on the worker host/container are not auto-propagated into job containers. Configure inside env on the pool template or deployment to ensure they’re present.
Docker job variable shapes (Prefect 3.x)
- `env`: dict of string key/values.
- `volumes`: list of Docker bind mount strings, e.g.:
- "/host/path:/container/path[:mode]" (e.g., :ro or :rw)
- Named volumes are fine: "my_named_vol:/container/path"
- You can also pass additional Docker options via container_create_kwargs (raw docker-py kwargs) or fields like network_mode, privileged, etc.
- To see the full schema Prefect expects, pull the default base job template:
- prefect work-pool get-default-base-job-template --type docker
All configuration options
1) Prefect UI (pool-level)
- Navigate to Work Pools -> your Docker pool -> “Edit base job template”
- Set env, volumes, network_mode, etc., and save
- Docs: Manage work pools
2) CLI (pool-level)
- Start from the default template, edit, then update the pool:
- View template:
prefect work-pool get-default-base-job-template --type docker > base.json
- Edit base.json to include your defaults (e.g., an env dict and volumes list)
- Create or update the pool with the template:
prefect work-pool create "my-docker-pool" --type docker --base-job-template ./base.json
# or, to update an existing pool
prefect work-pool update "my-docker-pool" --base-job-template ./base.json
- CLI reference: prefect work-pool CLI
Example snippet to put inside base.json (pool-level defaults):
{
"job_configuration": {
"image": "myrepo/myimage:latest",
"env": {
"SHARED_ENV": "value",
"LOG_LEVEL": "INFO"
},
"volumes": [
"/mnt/data:/data:rw",
"cache_volume:/cache"
],
"network_mode": "bridge",
"auto_remove": true
},
"variables": {}
}
Tip: It’s also valid to keep placeholders in job_configuration and set defaults in the variables section — use whichever you prefer.
3) Python SDK (pool-level)
- Programmatically create or update a pool’s base job template. Example:
```
import asyncio, json
from prefect.client.orchestration import get_client
async def main():
with open("base.json") as f:
template = json.load(f)
async with get_client() as client:Marvin
09/13/2025, 10:05 PMNote: The exact method signatures can vary slightly by version; see the API reference if you hit a mismatch. The key is passing `base_job_template` as a dict matching what `get-default-base-job-template` returns.
4) prefect.yaml (deployment-level)
- Set per-deployment `job_variables` to override or add to pool defaults:
deployments:
- name: etl-deployment
entrypoint: flows/etl.py:flow
work_pool:
name: my-docker-pool
job_variables:
env:
LOG_LEVEL: "DEBUG" # overrides pool default
EXTRA_FLAG: "1" # adds new var
volumes:
- "/mnt/data/datarw" # if you include volumes here, treat as the full desired list
- "extra_vol:/extra"
- Docs: <https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables|Customize job variables>
5) Python SDK (deployment-level)
- Override per deployment in code via `Flow.deploy`:
from prefect import flow
@flow
def my_flow():
...
my_flow.deploy(
name="etl-deployment",
work_pool_name="my-docker-pool",
job_variables={
"env": {"LOG_LEVEL": "DEBUG", "EXTRA_FLAG": "1"},
"volumes": ["/mnt/data/datarw", "extra_vol:/extra"],
},
)
6) Docker Compose (scripted pool-level)
- You can initialize or update the pool’s base template when your worker container starts:
services:
worker:
image: prefecthq/prefect:3-latest
environment:
PREFECT_API_URL: ${PREFECT_API_URL}
PREFECT_API_KEY: ${PREFECT_API_KEY}
volumes:
- ./base.json/config/base.jsonro
command: >
bash -lc "
prefect work-pool create my-docker-pool --type docker || true &&
prefect work-pool update my-docker-pool --base-job-template /config/base.json &&
prefect worker start -p my-docker-pool
"
```
- Important: env vars on the worker service above are not automatically forwarded to your flow-run containers. If you want them inside the job containers, set them in the pool base template or in deployment job_variables.
Useful references
- Manage work pools: docs
- Customize job variables: docs
- Work-pool CLI reference: docs
If you share how you prefer to manage these (UI-only, infra-as-code, etc.), I can tailor a minimal example for your setup.