<@ULVA73B9P> If i use work pools of type "docker" so that my deployed flows run inside of containers...
o
@Marvin If i use work pools of type "docker" so that my deployed flows run inside of containers created by the workers polling that pool, is it sufficient to bind/mount that volume at the worker pool level? Or is it necessary to include them in the deployment as job_variables? What about environment variables? If sufficient to set them at the work pool level, can i do so programmatically? (in a docker compose, prefect.yml, python SDK or similar). List all options please.
m
thought for 690.7 seconds
Short answer - Yes — for Docker work pools it’s sufficient to set mounts and environment variables on the work pool’s base job template; every container a worker launches from that pool will inherit them. - You only need to put them in a deployment’s
job_variables
if you want to override or add different values for that deployment. - Worker process environment variables are not automatically forwarded into the job container (except the Prefect-required ones like API URL/keys, etc.). If you want user-defined env vars inside the flow-run container, set them explicitly via the work pool base job template or deployment
job_variables
. - You can configure pool-level defaults via the UI, CLI, Python SDK, or by scripting (e.g., in Docker Compose) — all options below. Details and precedence - Where to set: - Work pool base job template: global defaults for all runs launched from that pool (recommended for mounts/env needed everywhere). - Deployment `job_variables`: per-deployment overrides or additions. - Precedence and merge: - Deployment
job_variables
override pool defaults. - For dict-like fields such as
env
, values are merged with deployment keys overriding duplicate keys. - For list-like fields such as
volumes
, treat the deployment-provided list as a replacement of the pool list. If you want “pool defaults + extra” volumes, include the entire final list at the deployment or keep all mounts at the pool level. - Automatic env propagation: - User-defined env on the worker host/container are not auto-propagated into job containers. Configure inside
env
on the pool template or deployment to ensure they’re present. Docker job variable shapes (Prefect 3.x) - `env`: dict of string key/values. - `volumes`: list of Docker bind mount strings, e.g.: -
"/host/path:/container/path[:mode]"
(e.g.,
:ro
or
:rw
) - Named volumes are fine:
"my_named_vol:/container/path"
- You can also pass additional Docker options via
container_create_kwargs
(raw docker-py kwargs) or fields like
network_mode
,
privileged
, etc. - To see the full schema Prefect expects, pull the default base job template: -
prefect work-pool get-default-base-job-template --type docker
All configuration options 1) Prefect UI (pool-level) - Navigate to Work Pools -> your Docker pool -> “Edit base job template” - Set
env
,
volumes
,
network_mode
, etc., and save - Docs: Manage work pools 2) CLI (pool-level) - Start from the default template, edit, then update the pool: - View template:
Copy code
prefect work-pool get-default-base-job-template --type docker > base.json
- Edit
base.json
to include your defaults (e.g., an
env
dict and
volumes
list) - Create or update the pool with the template:
Copy code
prefect work-pool create "my-docker-pool" --type docker --base-job-template ./base.json
    # or, to update an existing pool
    prefect work-pool update "my-docker-pool" --base-job-template ./base.json
- CLI reference: prefect work-pool CLI Example snippet to put inside base.json (pool-level defaults):
Copy code
{
  "job_configuration": {
    "image": "myrepo/myimage:latest",
    "env": {
      "SHARED_ENV": "value",
      "LOG_LEVEL": "INFO"
    },
    "volumes": [
      "/mnt/data:/data:rw",
      "cache_volume:/cache"
    ],
    "network_mode": "bridge",
    "auto_remove": true
  },
  "variables": {}
}
Tip: It’s also valid to keep placeholders in
job_configuration
and set defaults in the
variables
section — use whichever you prefer. 3) Python SDK (pool-level) - Programmatically create or update a pool’s base job template. Example: ``` import asyncio, json from prefect.client.orchestration import get_client async def main(): with open("base.json") as f: template = json.load(f) async with get_client() as client:
# Create (idempotent if the pool exists, update as needed) await client.create_work_pool( name="my-docker-pool", type="docker", base_job_template=template, ) # Or update an existing pool await client.update_work_pool( work_pool_name="my-docker-pool", base_job_template=template, ) asyncio.run(main())
Copy code
Note: The exact method signatures can vary slightly by version; see the API reference if you hit a mismatch. The key is passing `base_job_template` as a dict matching what `get-default-base-job-template` returns.

4) prefect.yaml (deployment-level)
- Set per-deployment `job_variables` to override or add to pool defaults:
deployments: - name: etl-deployment entrypoint: flows/etl.py:flow work_pool: name: my-docker-pool job_variables: env: LOG_LEVEL: "DEBUG" # overrides pool default EXTRA_FLAG: "1" # adds new var volumes: - "/mnt/data/datarw" # if you include volumes here, treat as the full desired list - "extra_vol:/extra"
Copy code
- Docs: <https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables|Customize job variables>

5) Python SDK (deployment-level)
- Override per deployment in code via `Flow.deploy`:
from prefect import flow @flow def my_flow(): ... my_flow.deploy( name="etl-deployment", work_pool_name="my-docker-pool", job_variables={ "env": {"LOG_LEVEL": "DEBUG", "EXTRA_FLAG": "1"}, "volumes": ["/mnt/data/datarw", "extra_vol:/extra"], }, )
Copy code
6) Docker Compose (scripted pool-level)
- You can initialize or update the pool’s base template when your worker container starts:
services: worker: image: prefecthq/prefect:3-latest environment: PREFECT_API_URL: ${PREFECT_API_URL} PREFECT_API_KEY: ${PREFECT_API_KEY} volumes: - ./base.json/config/base.jsonro command: > bash -lc " prefect work-pool create my-docker-pool --type docker || true && prefect work-pool update my-docker-pool --base-job-template /config/base.json && prefect worker start -p my-docker-pool " ``` - Important: env vars on the worker service above are not automatically forwarded to your flow-run containers. If you want them inside the job containers, set them in the pool base template or in deployment
job_variables
. Useful references - Manage work pools: docs - Customize job variables: docs - Work-pool CLI reference: docs If you share how you prefer to manage these (UI-only, infra-as-code, etc.), I can tailor a minimal example for your setup.