< Marvin> I get the follow log on a scheduled flow run Worke Prefect Community #ask-marvin

<@ULVA73B9P> I get the follow log on a scheduled f...

Olve Heitmann

11/26/2025, 11:06 PM

@Marvin I get the follow log on a scheduled flow run: Worker 'DockerWorker 608ed301-a0b0-47d2-86c2-5c62482e3063' submitting flow run 'fe298ebc-4cb5-4271-bd9b-8a08840615bd' 120412 AM prefect.flow_runs.worker Failed to submit flow run 'fe298ebc-4cb5-4271-bd9b-8a08840615bd' to infrastructure. Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/docker/api/client.py", line 275, in _raise_for_status response.raise_for_status() File "/usr/local/lib/python3.12/site-packages/requests/models.py", line 1026, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.52/containers/d25b175505ee5bd8961bfca3ffe17490197068bf25f3d64f9d1a948610c29877/start The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/prefect/workers/base.py", line 1271, in _submit_run_and_capture_errors result = await self.run( ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect_docker/worker.py", line 457, in run container, created_event = await run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 233, in run_sync_in_worker_thread result = await anyio.to_thread.run_sync( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2476, in run_sync_in_worker_thread return await future ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 967, in run result = context.run(func, *args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 243, in call_with_mark return call() ^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect_docker/worker.py", line 753, in _create_and_start_container container.start() File "/usr/local/lib/python3.12/site-packages/docker/models/containers.py", line 420, in start return self.client.api.start(self.id, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/docker/utils/decorators.py", line 19, in wrapped return f(self, resource_id, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/docker/api/container.py", line 1136, in start self._raise_for_status(res) File "/usr/local/lib/python3.12/site-packages/docker/api/client.py", line 277, in _raise_for_status raise create_api_error_from_http_exception(e) from e ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) from e docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.52/containers/d25b175505ee5bd8961bfca3ffe17490197068bf25f3d64f9d1a948610c29877/start: Not Found ("failed to set up container networking: network redpanda-local_redpanda_network not found") the error on redpanda-local_redpanda_network is not surprising, as the network have changed names due to changes in my docker compose to (qfin prefex rather than redpanda-local). but how can i make it detect the new one?

Marvin

11/26/2025, 11:06 PM

🔄 Researching your answer... this may take a while 🔧 Using tool:

research_prefect_topic

📊 Tools used: •

research_prefect_topic

(1x)

Olve Heitmann

11/26/2025, 11:34 PM

@Marvin did you freeze

Marvin

11/26/2025, 11:34 PM

✅ thought for 10.4 seconds

Marvin

11/26/2025, 11:34 PM

I’m here—sorry if it looked like I froze. Did you mean the chat, or is a Prefect flow/UI/worker hanging? If it’s Prefect, share your Prefect version and any run ID or logs and I’ll help you triage.

Olve Heitmann

11/26/2025, 11:35 PM

Marvin

11/26/2025, 11:35 PM

✅ thought for 401.3 seconds

Marvin

11/26/2025, 11:42 PM

You’re hitting it because the Docker worker is still trying to attach the container to the old network name. Prefect won’t auto-detect renamed Docker Compose networks—you need to update the

networks

setting wherever it’s defined. Where to look/update (Prefect 3.x): - Deployment job variables (per-deployment override) - Work pool’s base job template (pool-wide default) Quickly check where the old name is coming from: - Inspect the deployment:

Copy code

prefect deployment inspect "<flow-name>/<deployment-name>" --output json

Look under

job_variables.networks

(or

network_mode

if you use that). - Inspect the work pool:

Copy code

prefect work-pool inspect "<pool-name>" --output json

Look under

base_job_template.job_configuration.networks

. How to fix it Option A — Update the deployment’s job variables - In UI: Deployments > your deployment > Configuration > Job Variables > set

networks

to the new network name(s), then Save. - Via prefect.yaml (then redeploy):

Copy code

deployments:
  - name: your-deployment
    entrypoint: path/to/flow.py:flow
    work_pool:
      name: <pool-name>
      job_variables:
        networks:
          - qfin_<your_network_name>

- In code:

Copy code

flow.deploy(
  name="your-deployment",
  work_pool_name="<pool-name>",
  job_variables={"networks": ["qfin_<your_network_name>"]}
)

Option B — Update the work pool’s base job template (affects all deployments using this pool’s defaults)

Copy code

prefect work-pool inspect "<pool-name>" --output json > template.json
# edit template.json: set base_job_template.job_configuration.networks to ["qfin_<your_network_name>"]
prefect work-pool update "<pool-name>" --base-job-template template.json

Notes -

networks

and

network_mode

are mutually exclusive; set only one. - Compose project prefixes network names (e.g.,

qfin_xyz

). To avoid future renames, give your Compose network a stable name:

Copy code

networks:
  backend:
    name: backend        # stable name (no project prefix)
    driver: bridge

Then set

networks: ["backend"]

in Prefect. - On Linux, you can alternatively use

network_mode: "host"

if that fits your needs. If you share: - your work pool name, - the output of

prefect deployment inspect "<flow>/<deployment>" --output json

(job_variables section), - and the actual

docker network ls | grep qfin

network name, I can point you to the exact field to update.

Olve Heitmann

11/26/2025, 11:46 PM

@Marvin thing is I cant seem to find anywhere it is set in work pools nor deployment. Could it be baked into container runtime images?

Marvin

11/26/2025, 11:46 PM

✅ thought for 277.7 seconds

Marvin

11/26/2025, 11:51 PM

Short answer: no—Docker networks aren’t baked into images. They’re only applied at container runtime, and in Prefect they come from the resolved job configuration (deployment job_variables and/or the work pool’s base job template). Given your error includes a specific network name, the worker is receiving that name in its resolved job config. If you can’t see it in your deployment or pool at first glance, it’s likely one of these: - It’s hardcoded in the pool’s base job template under

job_configuration.networks

(not in

variables

), so it won’t show up as a variable. - It’s set as a default in the pool’s base job template under

variables.properties.networks.default

. - You’re inspecting a different pool than the one the run actually used. How to pinpoint exactly where it’s coming from 1) Inspect the failed run’s effective job config (most reliable) - In the UI: open the failed flow run, go to the Worker/Job details panel and view the “Job configuration” JSON. Look for

networks

network_mode

. 2) Confirm which pool the run used, then inspect that pool thoroughly - Get the pool name from the run or deployment. - Inspect the pool and check both job_configuration and variables defaults:

Copy code

prefect work-pool inspect "<pool-name>" --output json | jq '.base_job_template.job_configuration.networks // empty'
prefect work-pool inspect "<pool-name>" --output json | jq '.base_job_template.job_configuration.network_mode // empty'
prefect work-pool inspect "<pool-name>" --output json | jq '.base_job_template.variables.properties.networks.default // empty'

3) Inspect the deployment that scheduled the run

Copy code

prefect deployment inspect "<flow-name>/<deployment-name>" --output json | jq '.job_variables.networks // empty'
prefect deployment inspect "<flow-name>/<deployment-name>" --output json | jq '.job_variables.network_mode // empty'

Fix it once you find it - If set on the deployment, update its job variables to the new network:

Copy code

# in code
flow.deploy(
  name="<deployment>",
  work_pool_name="<pool>",
  job_variables={"networks": ["qfin_<new_network_name>"]}
)

- If set in the pool’s base job template, update it:

Copy code

prefect work-pool inspect "<pool>" --output json > template.json
# edit template.json: set base_job_template.job_configuration.networks to ["qfin_<new_network_name>"]
prefect work-pool update "<pool>" --base-job-template template.json

Tips -

networks

and

network_mode

are mutually exclusive; only set one. - If you want a stable name across Compose changes, give your Compose network an explicit

name:

so it doesn’t get a project prefix, and reference that in Prefect. If you share: - the pool name your failed run used, - the “Job configuration” JSON from that failed run, and/or - the outputs of the inspect commands above, I’ll point to the exact field to update.

Olve Heitmann

11/27/2025, 12:00 AM

@Marvin If I supply a network in the docker compose, is it necessary to also have it in job_configurations [networks] ?

Marvin

11/27/2025, 12:00 AM

✅ thought for 31.6 seconds

Marvin

11/27/2025, 12:01 AM

Short answer: not automatically. Defining a network in docker-compose does not make Prefect-launched flow containers join it. Unless you specify it in Prefect’s job configuration, the flow container will just use Docker’s default bridge network. When to set it: - If your flow container needs to talk to services on your compose network, add that network to Prefect: - Deployment job variables:

Copy code

work_pool:
    name: <pool>
    job_variables:
      networks: ["qfin_<your_network_name>"]

- Or set it pool-wide in the work pool’s base job template. - If you don’t need the custom network, remove any `networks`/`network_mode` from the deployment and pool; Prefect will stop trying to attach to the old network and run on bridge. Tips: - Compose often prefixes network names. Give the network a stable name to avoid future renames:

Copy code

networks:
    backend:
      name: backend

Then use

networks: ["backend"]

in Prefect. -

networks

and

network_mode

are mutually exclusive. On Linux,

network_mode: "host"

is an alternative if you need host access. If you tell me whether your flow needs to reach services on that compose network, I can suggest the minimal change (either set the network, or remove it everywhere).

2 Views

Open in Slack

Previous Next