<@ULVA73B9P> I have a prefect deployment script li...
# marvin-ai
k
@Marvin I have a prefect deployment script like this:
Copy code
from prefect.deployments import deploy
from prefect.deployments.runner import RunnerDeployment

deployment = RunnerDeployment(
    name="fraio-trigger",
    flow_name="trigger-flow",
    entrypoint="fraio_orchestrator.fraio_trigger_flow.trigger_flow",
    work_pool_name="fraio",
    job_variables={
        "env": {"PREFECT_API_URL": "<http://prefect:4200/api>"},
        "networks": ["agristoags-hbm_prefect-net"],
    },
    version="local",
    tags=[],
    description=None,
)

deploy(
    deployment,
    work_pool_name="fraio",
    image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
    build=False,
    push=False,
    print_next_steps_message=True,
    ignore_warnings=False,
)
now, that feels wrong, especially passing the env and networks job variables. I mean, when I run docker compose, I already pass the env variables to the docker worker and also define the network. I do understand that the flow run is in another docker container, but can't I just copy the network and env variables? The most annoying thing is that the network gets a specific name: if I call it
prefect-net
in docker compose, it becomes
<some-prefix>_prefect-net
. I don't want to bother with env vars and networks in the deployment script, but rather only in docker compose (for a docker worker which runs the flow runs) or in k8s settings somewhere (for a k8s worker).
m
🔄 Researching your answer... this may take a while 🔧 Using tool:
research_prefect_topic
📊 Tools used: •
display_callable_signature
(1x) •
explore_module_offerings
(1x) •
research_prefect_topic
(1x)
k
@Marvin did you get anywhere? thx.
m
âś… thought for 11.9 seconds
I don’t have the context handy—could you remind me what you’re waiting on? If you can share a quick recap (what you’re trying to do, what’s failing, any error snippets, and your flow/deployment/worker details), I’ll jump on it right away. If there’s a specific run, a link or run ID helps too.
k
@Marvin I have a prefect deployment script like this:
Copy code
from prefect.deployments import deploy
from prefect.deployments.runner import RunnerDeployment

deployment = RunnerDeployment(
    name="fraio-trigger",
    flow_name="trigger-flow",
    entrypoint="fraio_orchestrator.fraio_trigger_flow.trigger_flow",
    work_pool_name="fraio",
    job_variables={
        "env": {"PREFECT_API_URL": "<http://prefect:4200/api>"},
        "networks": ["agristoags-hbm_prefect-net"],
    },
    version="local",
    tags=[],
    description=None,
)

deploy(
    deployment,
    work_pool_name="fraio",
    image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
    build=False,
    push=False,
    print_next_steps_message=True,
    ignore_warnings=False,
)
now, that feels wrong, especially passing the env and networks job variables. I mean, when I run docker compose, I already pass the env variables to the docker worker and also define the network. I do understand that the flow run is in another docker container, but can't I just copy the network and env variables? The most annoying thing is that the network gets a specific name: if I call it
prefect-net
in docker compose, it becomes
<some-prefix>_prefect-net
. I don't want to bother with env vars and networks in the deployment script, but rather only in docker compose (for a docker worker which runs the flow runs) or in k8s settings somewhere (for a k8s worker).
âś… 1
m
âś… thought for 216.9 seconds
Short answer: you don’t need to put
env
and
networks
in every deployment. Put them once on the work pool’s base job template. Deployments can then omit those job_variables. Key points - Environment - Prefect settings from the worker process (e.g.,
PREFECT_API_URL
,
PREFECT_API_KEY
, etc.) are injected automatically into flow-run containers by the worker. You don’t need to repeat them in deployments as long as the worker is configured correctly. - Arbitrary extra env vars are not auto-copied from the worker’s container. If you want those for all runs, add them to the work pool’s base job template
env
, not per-deployment. - Network - Docker workers do not automatically “inherit” their own container network for flow-run containers. You must specify the network(s) for flow runs. The right place to do that is the work pool’s base job template, not each deployment. - Precedence (simplified) - Base job template defaults (work pool) → deployment
job_variables
override → flow-run
job_variables
override. For
env
, values are merged (last wins); most other fields are full overrides. How to set it up - Docker worker (recommended) - In the Prefect UI: Work Pools → your Docker pool (e.g., “fraio”) → Edit Base job template → set: - `networks`: ["agristoags-hbm_prefect-net"] (or your desired network) - `env`: any additional envs you want on all runs - Remove
env
and
networks
from your deployment script. - Kubernetes worker - Put your env/envFrom, resources, service account, etc., into the Kubernetes work pool’s base job template manifest. Again, remove from deployments. Dealing with Compose’s network name prefix - Compose changes
prefect-net
to
<project>_prefect-net
. Options: 1) Give the network a fixed name in compose:
Copy code
networks:
       default:
         name: prefect-net
2) Use an external network:
Copy code
# create once
     docker network create prefect-net

     # docker-compose.yml
     services:
       worker:
         networks: [prefect-net]
     networks:
       prefect-net:
         external: true
3) Set
COMPOSE_PROJECT_NAME
so the prefix is predictable and set that full name in the pool template. What your deployment can look like (cleaned)
Copy code
from prefect.deployments import deploy
from prefect.deployments.runner import RunnerDeployment

deployment = RunnerDeployment(
    name="fraio-trigger",
    flow_name="trigger-flow",
    entrypoint="fraio_orchestrator.fraio_trigger_flow.trigger_flow",
    work_pool_name="fraio",
    version="local",
    tags=[],
)

deploy(
    deployment,
    work_pool_name="fraio",
    image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
    build=False,
    push=False,
    print_next_steps_message=True,
)
Notes - If your
PREFECT_API_URL
inside the worker uses a Compose service name like
<http://prefect:4200/api>
, the flow-run container must be on the same user-defined network so the
prefect
DNS name resolves—hence setting
networks
on the pool template is the right fix. - You can still override per-deployment later if needed; the pool template gives you the sane defaults you’re looking for. Helpful links - Work pools and base job templates: https://docs.prefect.io/latest/concepts/work-pools/#base-job-template - Prefect Docker worker docs: https://prefecthq.github.io/prefect-docker/#docker-worker - Prefect Kubernetes worker docs: https://prefecthq.github.io/prefect-kubernetes/#kubernetes-worker If you want, share your current Docker pool’s base job template (or your Compose snippet), and I’ll suggest the exact template values so you can delete the `env`/`networks` from your deployments.
k
@Marvin so the flow-run container gets the same env vars and network as the work pool, right? Now, this is my workpool docker:
Copy code
prefect-workpool:
    build:
      context: .
      dockerfile: Dockerfile.prefect-workpool
    depends_on:
      prefect:
        condition: service_healthy
    environment:
      - PREFECT_API_URL=<http://prefect:4200/api>
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    entrypoint: ["/bin/bash", "-c"]
    command: |
      "
      set -e

      echo 'Creating work pool...'
      prefect work-pool create fraio -t docker --overwrite

      echo 'Starting worker for fraio...'
      prefect worker start --pool 'fraio'
      "
    networks: [prefect-net]
    profiles:
      - dev
having the dockerfile:
Copy code
FROM prefecthq/prefect:3.5.0-python3.12

# Install Docker CLI (system package)
RUN apt-get update \
        && apt-get install -y --no-install-recommends docker.io ca-certificates \
        && rm -rf /var/lib/apt/lists/* \
        # Install Prefect docker extras + python docker SDK
        && pip install --no-cache-dir --upgrade "prefect[docker]" docker
This means the flow-worker should run with env var
PREFECT_API_URL
from this container (unless overwritten) and the same network as this work pool?
m
âś… thought for 65.1 seconds
Almost: env, yes; network, no. - Env vars: Prefect settings from the worker process (like
PREFECT_API_URL
) are injected into flow-run containers automatically by the Docker worker. Since your worker container sets
PREFECT_API_URL=<http://prefect:4200/api>
, flow runs will inherit it unless you override
env
in job variables. You do not need to repeat this in every deployment. - Network: The flow-run container does not automatically inherit the worker container’s Docker network. You must specify the network(s) for flow runs in the Docker work pool’s base job template. Once set there, deployments can omit
networks
. What to do - In the UI: Work Pools → fraio → Edit base job template → set: - `networks`: ["prefect-net"] (or the exact Compose network name) - Optionally add any global
env
you want on all runs (not needed for
PREFECT_API_URL
since the worker already provides it via Prefect settings) Avoiding the Compose network prefix - Compose often renames networks to
<project>_prefect-net
. Use one of: - Give the network a fixed name in compose:
Copy code
networks:
      default:
        name: prefect-net
- Or make it external:
Copy code
# create once
    docker network create prefect-net

    # docker-compose.yml
    services:
      prefect-workpool:
        networks: [prefect-net]
    networks:
      prefect-net:
        external: true
- Or set
COMPOSE_PROJECT_NAME
so the full name is predictable and use that full name in the pool template. Why this matters for your setup - Your
PREFECT_API_URL
uses the Compose service name
prefect
. For that name to resolve inside the flow-run container, the flow-run container must be attached to the same user-defined network as the
prefect
service—hence configuring
networks
on the work pool’s base job template is the right fix. References - Work pools and base job templates: https://docs.prefect.io/latest/concepts/work-pools/#base-job-template - Docker worker docs: https://prefecthq.github.io/prefect-docker/#docker-worker If you share your current base job template (from the UI “Inspect” view), I can suggest the exact JSON to add so you can remove `env`/`networks` from deployments.
k
@Marvin 1. right, but can I do it from the api instead of from the UI? 2. The "Docker work pool’s base job template" is basically this?
Copy code
deployment = RunnerDeployment(
    name="fraio-trigger",
    flow_name="trigger-flow",
    entrypoint="fraio_orchestrator.fraio_trigger_flow.trigger_flow",
    work_pool_name="fraio",
    job_variables={
        "env": {"PREFECT_API_URL": "<http://prefect:4200/api>"},
        "networks": ["agristoags-hbm_prefect-net"],
    },
    version="local",
    tags=[],
    description=None,
)

deploy(
    deployment,
    work_pool_name="fraio",
    image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
    build=False,
    push=False,
    print_next_steps_message=True,
    ignore_warnings=False,
)
m
âś… thought for 155.9 seconds
Great questions—two parts: 1) Can I set the work pool base job template via API instead of the UI? Yes. Use the Prefect client to update the work pool’s
base_job_template
. Example (sync):
Copy code
from prefect.client.orchestration import get_client
from prefect.client.schemas.actions import WorkPoolUpdate

POOL_NAME = "fraio"

with get_client(sync_client=True) as client:
    wp = client.read_work_pool(POOL_NAME)
    tmpl = dict(wp.base_job_template or {})
    vars = dict(tmpl.get("variables", {}))

    # Set default Docker networks for all flow-run containers in this pool
    vars["networks"] = {
        **vars.get("networks", {}),
        "default": ["agristoags-hbm_prefect-net"],
    }

    # Optional: global env for all runs (you usually do NOT need PREFECT_API_URL here)
    # vars["env"] = {
    #     **vars.get("env", {}),
    #     "default": {"FOO": "bar"}
    # }

    tmpl["variables"] = vars

    client.update_work_pool(
        work_pool_name=POOL_NAME,
        work_pool=WorkPoolUpdate(base_job_template=tmpl),
    )
    print("Updated work pool base job template")
You can run that as a one-off initializer in your worker container (e.g., right after creating the work pool) before starting the worker:
Copy code
prefect work-pool create fraio -t docker --overwrite
python - <<'PY'
# paste the Python snippet here
PY
prefect worker start --pool fraio
Notes: - You do not need to set
PREFECT_API_URL
in the base job template `env`; since it’s a Prefect setting on the worker process, the Docker worker injects it into flow-run containers automatically unless you explicitly override it in job variables. - Networks must match the actual Docker network name. If Compose prefixes it, consider a fixed/external network name to avoid surprises. 2) Is the “Docker work pool’s base job template” basically the
job_variables
in my deployment snippet? No. They’re different layers: - Work pool base job template = global defaults for all runs in that pool (what you want for networks and any global env). - Deployment
job_variables
= per-deployment overrides of the base job template. In your snippet, you’re overriding at the deployment level. If you move
networks
(and any global env) into the pool’s base job template as shown above, you can remove them from the deployment:
Copy code
from prefect.deployments import deploy
from prefect.deployments.runner import RunnerDeployment

deployment = RunnerDeployment(
    name="fraio-trigger",
    flow_name="trigger-flow",
    entrypoint="fraio_orchestrator.fraio_trigger_flow.trigger_flow",
    work_pool_name="fraio",
    version="local",
    tags=[],
)

deploy(
    deployment,
    work_pool_name="fraio",
    image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
    build=False,
    push=False,
    print_next_steps_message=True,
)
Recap for your setup - Keep
PREFECT_API_URL=<http://prefect:4200/api>
on the worker container env (it will be injected automatically). - Put
networks: ["agristoags-hbm_prefect-net"]
into the fraio work pool’s base job template so every flow-run container lands on the same network as your
prefect
service. - Remove `env`/`networks` from deployments unless you need per-deployment overrides. Docs for reference - Work pools and base job templates: https://docs.prefect.io/latest/concepts/work-pools/#base-job-template - Docker worker: https://prefecthq.github.io/prefect-docker/#docker-worker If you want, share the current
wp.base_job_template
(from client.read_work_pool("fraio")) and the Compose network config; I can suggest the exact JSON to set so you can delete `env`/`networks` from deployments.
k
@Marvin right, but I can't define the default network (base template) that inside this docker command?
Copy code
prefect-workpool:
    build:
      context: .
      dockerfile: Dockerfile.prefect-workpool
    depends_on:
      prefect:
        condition: service_healthy
    environment:
      - PREFECT_API_URL=<http://prefect:4200/api>
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    entrypoint: ["/bin/bash", "-c"]
    command: |
      "
      set -e

      echo 'Creating work pool...'
      prefect work-pool create fraio -t docker --overwrite

      echo 'Starting worker for fraio...'
      prefect worker start --pool 'fraio'
      "
    networks: [prefect-net]
    profiles:
      - dev
m
âś… thought for 86.4 seconds
Yes—you can set the pool’s default network from that command block. You just need to pass a base job template to the CLI when you create/update the work pool. There isn’t a
--networks
flag; you provide JSON via
--base-job-template
. Option A: Inline JSON (here-doc) in your Compose command
Copy code
entrypoint: ["/bin/bash", "-c"]
command: |
  "
  set -e

  echo 'Creating work pool...'
  cat >/tmp/fraio-base.json <<'JSON'
  {
    "variables": {
      "networks": { "default": ["agristoags-hbm_prefect-net"] }
      // Optional: env defaults for all runs
      // "env": { "default": { "FOO": "bar" } }
    }
  }
  JSON

  prefect work-pool create fraio -t docker --base-job-template /tmp/fraio-base.json --overwrite

  echo 'Starting worker for fraio...'
  prefect worker start --pool 'fraio'
  "
Option B: Patch the default template with jq
Copy code
prefect work-pool get-default-base-job-template --type docker > /tmp/base.json
jq '.variables.networks.default=["agristoags-hbm_prefect-net"]' /tmp/base.json > /tmp/base.updated.json
prefect work-pool create fraio -t docker --base-job-template /tmp/base.updated.json --overwrite
prefect worker start --pool fraio
Option C: Use the Python client inline
Copy code
python - <<'PY'
from prefect.client.orchestration import get_client
from prefect.client.schemas.actions import WorkPoolUpdate
POOL = "fraio"
with get_client(sync_client=True) as client:
    wp = client.read_work_pool(POOL)
    tmpl = dict(wp.base_job_template or {})
    vars = dict(tmpl.get("variables", {}))
    vars["networks"] = {"default": ["agristoags-hbm_prefect-net"]}
    # Optional: env defaults
    # vars["env"] = {"default": {"FOO": "bar"}}
    tmpl["variables"] = vars
    client.update_work_pool(POOL, WorkPoolUpdate(base_job_template=tmpl))
PY
prefect worker start --pool fraio
Notes - You don’t need to set
PREFECT_API_URL
in the base template; the Docker worker injects Prefect settings from the worker process automatically. - The worker’s container network is not auto-inherited by flow containers; setting
variables.networks.default
in the pool template is the right way. - If Compose prefixes your network name, consider using a fixed or external network name to avoid surprises. CLI references - Create/update with a base job template file: https://docs.prefect.io/latest/concepts/work-pools/#base-job-template - CLI help: run
prefect work-pool create --help
and
prefect work-pool update --help
in your container.