<@ULVA73B9P> I'm trying to set extra_hosts on a do...
# ask-marvin
n
@Marvin I'm trying to set extra_hosts on a docker container but the extra hosts stays empty:
Copy code
"container_create_kwargs": {
    "extra_hosts": {
      "a": "ip"
    }
  }
Prefect version 3.6.13
m
thought for 209.8 seconds
Thanks for the details. In Prefect 3.6.13 with the Docker work pool/worker,
extra_hosts
should be passed via
container_create_kwargs
in the job variables (deployment) or in the work pool’s base job template. If it’s “staying empty”, it’s usually one of these: - The setting is in the wrong place (not under
work_pool.job_variables.container_create_kwargs
) - A base job template is overriding it - The worker is not picking up the latest deployment/job variables - The value is being dropped during merge because it’s not in the expected shape A quick checklist and a working example: 1) Put it under job variables for your deployment (correct nesting)
Copy code
deployments:
  - name: my-deployment
    entrypoint: flow.py:my_flow
    work_pool:
      name: my-docker-pool
      job_variables:
        container_create_kwargs:
          extra_hosts:
            a: "1.2.3.4"
Or programmatically:
Copy code
flow.from_source(...).deploy(
  name="my-deployment",
  work_pool_name="my-docker-pool",
  job_variables={
    "container_create_kwargs": {
      "extra_hosts": {
        "a": "1.2.3.4"
      }
    }
  }
)
2) Valid shapes for extra_hosts - Dict (recommended):
{"a": "1.2.3.4"}
- Or list of strings:
["a:1.2.3.4"]
Your dict form is correct; just be sure the IP is a valid IP address. 3) Verify what the deployment actually has Run:
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
and check the
work_pool.job_variables.container_create_kwargs.extra_hosts
section. 4) Verify in the worker/container - In the Docker worker logs (set worker log level to DEBUG if possible), look for the final resolved
container_create_kwargs
. - On a run container, check:
Copy code
docker inspect <container_id> | jq '.[0].HostConfig.ExtraHosts'
You should see something like:
Copy code
["a:1.2.3.4", "host.docker.internal:host-gateway"]  # on Linux
Notes: - On Linux, the Docker worker will also add
host.docker.internal:host-gateway
automatically; your entries should merge with that (not overwrite). - Make sure you’re using a Docker work pool/worker (not Kubernetes), and that the worker process is running with a version of the
prefect-docker
integration that matches your Prefect version (e.g.,
pip show prefect-docker
in the worker environment). If it’s still empty, could you share: - The output of
prefect deployment inspect "<flow>/<deployment>"
(just the
work_pool.job_variables
portion) - The Docker worker logs around container creation - The result of
pip show prefect-docker
from the worker environment That’ll help pinpoint whether it’s a placement/merge issue or something environmental.
n
@Marvin How do I know if "The worker is not picking up the latest deployment/job variables"?
m
thought for 102.6 seconds
Great question. In Prefect 3.x, you don’t need to restart workers, but here’s how to tell whether your worker has the latest settings and is actually using them. What updates when - Deployment job_variables: Fetched fresh for every submitted flow run. Any change you make is used on the next run (no heartbeat/restart needed). - Work pool base job template: The worker refreshes this on its next heartbeat (about every 30s by default). Changes take effect for runs submitted after that sync. How to verify 1) Confirm the backend has your latest values - Deployment:
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
Check
work_pool.job_variables.container_create_kwargs.extra_hosts
. - Work pool:
Copy code
prefect work-pool inspect "<your-work-pool>"
Check the base job template for
container_create_kwargs.extra_hosts
. 2) Watch the worker pick up changes - Check the worker logs after you update the deployment/job template. You should see a heartbeat/sync within ~30s. The next submitted run will use the updated template/variables. - Run another flow and then inspect the created container:
Copy code
docker inspect <container_id> | jq '.[0].HostConfig.ExtraHosts'
You should see your entries (and possibly
host.docker.internal:host-gateway
on Linux). 3) Force a quick confirmation - After updating the deployment/job variables, immediately trigger a new run. If the container still lacks your hosts, it means the worker used older config. Wait ~30s after changing the work pool template (for heartbeat), then trigger another run and re-check. If it still looks stale - Ensure you edited the right place (deployment job_variables vs work pool template) and the nesting is correct under
work_pool.job_variables.container_create_kwargs.extra_hosts
. - Make sure the worker is connected to the same workspace and work pool you updated. - Consider enabling more verbose logs on the worker to see the resolved configuration just before container creation. If you share: - The output of
prefect deployment inspect "<flow>/<deployment>"
(job_variables portion) -
prefect work-pool inspect "<pool>"
- A snippet of the worker logs around a submission I can help pinpoint exactly where it’s getting out of sync.