Kurt Sys (Vintecc)
12/08/2025, 6:29 PMfrom prefect.deployments import deploy
from prefect.deployments.runner import RunnerDeployment
deployment = RunnerDeployment(
name="fraio-trigger",
flow_name="trigger-flow",
entrypoint="fraio_orchestrator.fraio_trigger_flow.trigger_flow",
work_pool_name="fraio",
job_variables={
"env": {"PREFECT_API_URL": "<http://prefect:4200/api>"},
"networks": ["agristoags-hbm_prefect-net"],
},
version="local",
tags=[],
description=None,
)
deploy(
deployment,
work_pool_name="fraio",
image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
build=False,
push=False,
print_next_steps_message=True,
ignore_warnings=False,
)
now, that feels wrong, especially passing the env and networks job variables. I mean, when I run docker compose, I already pass the env variables to the docker worker and also define the network. I do understand that the flow run is in another docker container, but can't I just copy the network and env variables? The most annoying thing is that the network gets a specific name: if I call it prefect-net in docker compose, it becomes <some-prefix>_prefect-net. I don't want to bother with env vars and networks in the deployment script, but rather only in docker compose (for a docker worker which runs the flow runs) or in k8s settings somewhere (for a k8s worker).Marvin
12/08/2025, 6:29 PMresearch_prefect_topic
📊 Tools used:
• display_callable_signature (1x)
• explore_module_offerings (1x)
• research_prefect_topic (1x)Kurt Sys (Vintecc)
12/09/2025, 7:33 AMMarvin
12/09/2025, 7:33 AMMarvin
12/09/2025, 7:33 AMKurt Sys (Vintecc)
12/09/2025, 7:35 AMfrom prefect.deployments import deploy
from prefect.deployments.runner import RunnerDeployment
deployment = RunnerDeployment(
name="fraio-trigger",
flow_name="trigger-flow",
entrypoint="fraio_orchestrator.fraio_trigger_flow.trigger_flow",
work_pool_name="fraio",
job_variables={
"env": {"PREFECT_API_URL": "<http://prefect:4200/api>"},
"networks": ["agristoags-hbm_prefect-net"],
},
version="local",
tags=[],
description=None,
)
deploy(
deployment,
work_pool_name="fraio",
image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
build=False,
push=False,
print_next_steps_message=True,
ignore_warnings=False,
)
now, that feels wrong, especially passing the env and networks job variables. I mean, when I run docker compose, I already pass the env variables to the docker worker and also define the network. I do understand that the flow run is in another docker container, but can't I just copy the network and env variables? The most annoying thing is that the network gets a specific name: if I call it prefect-net in docker compose, it becomes <some-prefix>_prefect-net. I don't want to bother with env vars and networks in the deployment script, but rather only in docker compose (for a docker worker which runs the flow runs) or in k8s settings somewhere (for a k8s worker).Marvin
12/09/2025, 7:35 AMMarvin
12/09/2025, 7:38 AMenv and networks in every deployment. Put them once on the work pool’s base job template. Deployments can then omit those job_variables.
Key points
- Environment
- Prefect settings from the worker process (e.g., PREFECT_API_URL, PREFECT_API_KEY, etc.) are injected automatically into flow-run containers by the worker. You don’t need to repeat them in deployments as long as the worker is configured correctly.
- Arbitrary extra env vars are not auto-copied from the worker’s container. If you want those for all runs, add them to the work pool’s base job template env, not per-deployment.
- Network
- Docker workers do not automatically “inherit” their own container network for flow-run containers. You must specify the network(s) for flow runs. The right place to do that is the work pool’s base job template, not each deployment.
- Precedence (simplified)
- Base job template defaults (work pool) → deployment job_variables override → flow-run job_variables override. For env, values are merged (last wins); most other fields are full overrides.
How to set it up
- Docker worker (recommended)
- In the Prefect UI: Work Pools → your Docker pool (e.g., “fraio”) → Edit Base job template → set:
- `networks`: ["agristoags-hbm_prefect-net"] (or your desired network)
- `env`: any additional envs you want on all runs
- Remove env and networks from your deployment script.
- Kubernetes worker
- Put your env/envFrom, resources, service account, etc., into the Kubernetes work pool’s base job template manifest. Again, remove from deployments.
Dealing with Compose’s network name prefix
- Compose changes prefect-net to <project>_prefect-net. Options:
1) Give the network a fixed name in compose:
networks:
default:
name: prefect-net
2) Use an external network:
# create once
docker network create prefect-net
# docker-compose.yml
services:
worker:
networks: [prefect-net]
networks:
prefect-net:
external: true
3) Set COMPOSE_PROJECT_NAME so the prefix is predictable and set that full name in the pool template.
What your deployment can look like (cleaned)
from prefect.deployments import deploy
from prefect.deployments.runner import RunnerDeployment
deployment = RunnerDeployment(
name="fraio-trigger",
flow_name="trigger-flow",
entrypoint="fraio_orchestrator.fraio_trigger_flow.trigger_flow",
work_pool_name="fraio",
version="local",
tags=[],
)
deploy(
deployment,
work_pool_name="fraio",
image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
build=False,
push=False,
print_next_steps_message=True,
)
Notes
- If your PREFECT_API_URL inside the worker uses a Compose service name like <http://prefect:4200/api>, the flow-run container must be on the same user-defined network so the prefect DNS name resolves—hence setting networks on the pool template is the right fix.
- You can still override per-deployment later if needed; the pool template gives you the sane defaults you’re looking for.
Helpful links
- Work pools and base job templates: https://docs.prefect.io/latest/concepts/work-pools/#base-job-template
- Prefect Docker worker docs: https://prefecthq.github.io/prefect-docker/#docker-worker
- Prefect Kubernetes worker docs: https://prefecthq.github.io/prefect-kubernetes/#kubernetes-worker
If you want, share your current Docker pool’s base job template (or your Compose snippet), and I’ll suggest the exact template values so you can delete the `env`/`networks` from your deployments.Kurt Sys (Vintecc)
12/09/2025, 7:41 AMprefect-workpool:
build:
context: .
dockerfile: Dockerfile.prefect-workpool
depends_on:
prefect:
condition: service_healthy
environment:
- PREFECT_API_URL=<http://prefect:4200/api>
volumes:
- /var/run/docker.sock:/var/run/docker.sock
entrypoint: ["/bin/bash", "-c"]
command: |
"
set -e
echo 'Creating work pool...'
prefect work-pool create fraio -t docker --overwrite
echo 'Starting worker for fraio...'
prefect worker start --pool 'fraio'
"
networks: [prefect-net]
profiles:
- dev
having the dockerfile:
FROM prefecthq/prefect:3.5.0-python3.12
# Install Docker CLI (system package)
RUN apt-get update \
&& apt-get install -y --no-install-recommends docker.io ca-certificates \
&& rm -rf /var/lib/apt/lists/* \
# Install Prefect docker extras + python docker SDK
&& pip install --no-cache-dir --upgrade "prefect[docker]" docker
This means the flow-worker should run with env var PREFECT_API_URL from this container (unless overwritten) and the same network as this work pool?Marvin
12/09/2025, 7:42 AMMarvin
12/09/2025, 7:43 AMPREFECT_API_URL) are injected into flow-run containers automatically by the Docker worker. Since your worker container sets PREFECT_API_URL=<http://prefect:4200/api>, flow runs will inherit it unless you override env in job variables. You do not need to repeat this in every deployment.
- Network: The flow-run container does not automatically inherit the worker container’s Docker network. You must specify the network(s) for flow runs in the Docker work pool’s base job template. Once set there, deployments can omit networks.
What to do
- In the UI: Work Pools → fraio → Edit base job template → set:
- `networks`: ["prefect-net"] (or the exact Compose network name)
- Optionally add any global env you want on all runs (not needed for PREFECT_API_URL since the worker already provides it via Prefect settings)
Avoiding the Compose network prefix
- Compose often renames networks to <project>_prefect-net. Use one of:
- Give the network a fixed name in compose:
networks:
default:
name: prefect-net
- Or make it external:
# create once
docker network create prefect-net
# docker-compose.yml
services:
prefect-workpool:
networks: [prefect-net]
networks:
prefect-net:
external: true
- Or set COMPOSE_PROJECT_NAME so the full name is predictable and use that full name in the pool template.
Why this matters for your setup
- Your PREFECT_API_URL uses the Compose service name prefect. For that name to resolve inside the flow-run container, the flow-run container must be attached to the same user-defined network as the prefect service—hence configuring networks on the work pool’s base job template is the right fix.
References
- Work pools and base job templates: https://docs.prefect.io/latest/concepts/work-pools/#base-job-template
- Docker worker docs: https://prefecthq.github.io/prefect-docker/#docker-worker
If you share your current base job template (from the UI “Inspect” view), I can suggest the exact JSON to add so you can remove `env`/`networks` from deployments.Kurt Sys (Vintecc)
12/09/2025, 7:45 AMdeployment = RunnerDeployment(
name="fraio-trigger",
flow_name="trigger-flow",
entrypoint="fraio_orchestrator.fraio_trigger_flow.trigger_flow",
work_pool_name="fraio",
job_variables={
"env": {"PREFECT_API_URL": "<http://prefect:4200/api>"},
"networks": ["agristoags-hbm_prefect-net"],
},
version="local",
tags=[],
description=None,
)
deploy(
deployment,
work_pool_name="fraio",
image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
build=False,
push=False,
print_next_steps_message=True,
ignore_warnings=False,
)Marvin
12/09/2025, 7:48 AMMarvin
12/09/2025, 7:51 AMbase_job_template. Example (sync):
from prefect.client.orchestration import get_client
from prefect.client.schemas.actions import WorkPoolUpdate
POOL_NAME = "fraio"
with get_client(sync_client=True) as client:
wp = client.read_work_pool(POOL_NAME)
tmpl = dict(wp.base_job_template or {})
vars = dict(tmpl.get("variables", {}))
# Set default Docker networks for all flow-run containers in this pool
vars["networks"] = {
**vars.get("networks", {}),
"default": ["agristoags-hbm_prefect-net"],
}
# Optional: global env for all runs (you usually do NOT need PREFECT_API_URL here)
# vars["env"] = {
# **vars.get("env", {}),
# "default": {"FOO": "bar"}
# }
tmpl["variables"] = vars
client.update_work_pool(
work_pool_name=POOL_NAME,
work_pool=WorkPoolUpdate(base_job_template=tmpl),
)
print("Updated work pool base job template")
You can run that as a one-off initializer in your worker container (e.g., right after creating the work pool) before starting the worker:
prefect work-pool create fraio -t docker --overwrite
python - <<'PY'
# paste the Python snippet here
PY
prefect worker start --pool fraio
Notes:
- You do not need to set PREFECT_API_URL in the base job template `env`; since it’s a Prefect setting on the worker process, the Docker worker injects it into flow-run containers automatically unless you explicitly override it in job variables.
- Networks must match the actual Docker network name. If Compose prefixes it, consider a fixed/external network name to avoid surprises.
2) Is the “Docker work pool’s base job template” basically the job_variables in my deployment snippet?
No. They’re different layers:
- Work pool base job template = global defaults for all runs in that pool (what you want for networks and any global env).
- Deployment job_variables = per-deployment overrides of the base job template.
In your snippet, you’re overriding at the deployment level. If you move networks (and any global env) into the pool’s base job template as shown above, you can remove them from the deployment:
from prefect.deployments import deploy
from prefect.deployments.runner import RunnerDeployment
deployment = RunnerDeployment(
name="fraio-trigger",
flow_name="trigger-flow",
entrypoint="fraio_orchestrator.fraio_trigger_flow.trigger_flow",
work_pool_name="fraio",
version="local",
tags=[],
)
deploy(
deployment,
work_pool_name="fraio",
image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
build=False,
push=False,
print_next_steps_message=True,
)
Recap for your setup
- Keep PREFECT_API_URL=<http://prefect:4200/api> on the worker container env (it will be injected automatically).
- Put networks: ["agristoags-hbm_prefect-net"] into the fraio work pool’s base job template so every flow-run container lands on the same network as your prefect service.
- Remove `env`/`networks` from deployments unless you need per-deployment overrides.
Docs for reference
- Work pools and base job templates: https://docs.prefect.io/latest/concepts/work-pools/#base-job-template
- Docker worker: https://prefecthq.github.io/prefect-docker/#docker-worker
If you want, share the current wp.base_job_template (from client.read_work_pool("fraio")) and the Compose network config; I can suggest the exact JSON to set so you can delete `env`/`networks` from deployments.Kurt Sys (Vintecc)
12/09/2025, 8:25 AMprefect-workpool:
build:
context: .
dockerfile: Dockerfile.prefect-workpool
depends_on:
prefect:
condition: service_healthy
environment:
- PREFECT_API_URL=<http://prefect:4200/api>
volumes:
- /var/run/docker.sock:/var/run/docker.sock
entrypoint: ["/bin/bash", "-c"]
command: |
"
set -e
echo 'Creating work pool...'
prefect work-pool create fraio -t docker --overwrite
echo 'Starting worker for fraio...'
prefect worker start --pool 'fraio'
"
networks: [prefect-net]
profiles:
- devMarvin
12/09/2025, 8:26 AMMarvin
12/09/2025, 8:27 AM--networks flag; you provide JSON via --base-job-template.
Option A: Inline JSON (here-doc) in your Compose command
entrypoint: ["/bin/bash", "-c"]
command: |
"
set -e
echo 'Creating work pool...'
cat >/tmp/fraio-base.json <<'JSON'
{
"variables": {
"networks": { "default": ["agristoags-hbm_prefect-net"] }
// Optional: env defaults for all runs
// "env": { "default": { "FOO": "bar" } }
}
}
JSON
prefect work-pool create fraio -t docker --base-job-template /tmp/fraio-base.json --overwrite
echo 'Starting worker for fraio...'
prefect worker start --pool 'fraio'
"
Option B: Patch the default template with jq
prefect work-pool get-default-base-job-template --type docker > /tmp/base.json
jq '.variables.networks.default=["agristoags-hbm_prefect-net"]' /tmp/base.json > /tmp/base.updated.json
prefect work-pool create fraio -t docker --base-job-template /tmp/base.updated.json --overwrite
prefect worker start --pool fraio
Option C: Use the Python client inline
python - <<'PY'
from prefect.client.orchestration import get_client
from prefect.client.schemas.actions import WorkPoolUpdate
POOL = "fraio"
with get_client(sync_client=True) as client:
wp = client.read_work_pool(POOL)
tmpl = dict(wp.base_job_template or {})
vars = dict(tmpl.get("variables", {}))
vars["networks"] = {"default": ["agristoags-hbm_prefect-net"]}
# Optional: env defaults
# vars["env"] = {"default": {"FOO": "bar"}}
tmpl["variables"] = vars
client.update_work_pool(POOL, WorkPoolUpdate(base_job_template=tmpl))
PY
prefect worker start --pool fraio
Notes
- You don’t need to set PREFECT_API_URL in the base template; the Docker worker injects Prefect settings from the worker process automatically.
- The worker’s container network is not auto-inherited by flow containers; setting variables.networks.default in the pool template is the right way.
- If Compose prefixes your network name, consider using a fixed or external network name to avoid surprises.
CLI references
- Create/update with a base job template file: https://docs.prefect.io/latest/concepts/work-pools/#base-job-template
- CLI help: run prefect work-pool create --help and prefect work-pool update --help in your container.