<@ULVA73B9P> how to stop crashing? When I run it a...
# ask-marvin
f
@Marvin how to stop crashing? When I run it again it works alright. docker.errors.ImageNotFound: 404 Client Error for http+docker://localnpipe/v1.49/images/create?tag=local&amp;fromImage=data_unload_local: Not Found ("pull access denied for data_unload_local, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
m
Error: No module named 'prefect.workers.docker'
thought for 267.0 seconds
Short answer: something in your run is trying to PULL an image called data_unload_local:local from a registry. It doesn’t exist there, so Docker says “denied”. On the next attempt it probably finds the image locally (or your build completed by then), so it works. To stop the crash, either make the image available in a registry or configure your stack to not pull when you’re using a local-only image. A few concrete paths forward (pick the one that matches how you’re running): - If you’re running containers directly with docker-py (Docker SDK): - Don’t call client.images.pull(...) for a local-only image. Instead: - Try client.images.get("data_unload_local:local"); if that raises, build or tag it first. - Only pull if you truly expect it to be in a registry. - Make sure you always specify repo:tag exactly: data_unload_local:local - If you’re running on Kubernetes: - Set imagePullPolicy to IfNotPresent (or Never) in your Job/Pod template so the node uses the local image if it exists and doesn’t try to pull. - If you’re using Prefect 3.x deployments: - Using a local image without pushing: - Build/tag the image locally first: docker build -t data_unload_local:local . - Deploy specifying your local image and avoid pushing: - CLI: prefect deploy ... --image data_unload_local:local --push False - Or via Python deploy(image="data_unload_local:local", build=False, push=False) - Make sure the worker/executor runs on the same machine (same Docker daemon) where the image is present. If the worker is on a different host, your local image won’t exist there and it will try to pull (and fail). - Building a dev image: - prefect dev build-image builds a Prefect base image for development; you can extend it for your flow, then tag as data_unload_local:local - Help: prefect dev build-image --help - Prefect docs for Docker-based deployment considerations: - https://docs-3.prefect.io/v3/how-to-guides/deployment_infra/docker - If you actually want to pull from a registry: - Push the image to a registry (or use a local registry like localhost:5000), tag it accordingly (e.g., localhost5000/data unload locallocal), docker login if needed, then reference that fully qualified name in your run. Why it’s intermittent - Race conditions are common: the first run tries to pull (fails), meanwhile your build or tag completes; the second run no longer pulls and finds the image locally. - Different Docker daemons: if your first attempt hits a daemon without the image (e.g., a remote worker), it pulls (and fails); the second attempt runs on your local daemon where the image exists. If you can share: - How you launch the container (Prefect version, worker type, Kubernetes vs Docker SDK vs something else) - The exact image string you pass - Whether your worker runs on the same machine as your image build I can give you the precise configuration/command to stop the pulls for your setup. I’d love to say it’ll be exciting, but really it’ll just stop crashing.
f
@Marvin I m using this to deploy: .deploy( name=flow_name, work_pool_name="docker-worker-pool", image=DockerImage( name=flow_name, tag=environment, dockerfile="data_flow.Dockerfile", rm=True # Removes intermediate containers ), push=False, tags=tags_for_deployment, job_variables = { "env": { "ENVIRONMENT": environment, "PROJECT_NAME" : project_name # Add more variables here, separated by commas... } } )
m
thought for 174.0 seconds
Thanks—that snippet explains the behavior. You’re building an image locally during deploy, but push=False means the image never makes it to the Docker worker’s host. When the worker tries to start the flow, it can’t find the image locally, so it attempts to pull it from a registry. Since your image name is just flow_name:environment (e.g., data_unload_local:local) and not in any registry, the pull fails with “access denied.” On the next run, you sometimes get lucky (e.g., same daemon, image now present), so it works. Computers are nothing if not consistent—except when they aren’t. You’ve got two clean ways to stop the crash—pick one: Option A: Use a registry (recommended if the worker isn’t on the same machine) - Tag for a registry and push the image during deploy: - Example: image=DockerImage( name="ghcr.io/your-org/data_unload", # or ECR/GCR/ACR/Docker Hub namespace tag=environment, dockerfile="data_flow.Dockerfile", rm=True ) .deploy(..., push=True, ...) - Ensure the worker can pull: - docker login on the worker host (or configure credentials in whatever way you normally do) - Use the fully qualified image name above in your deployment - Docs: Docker deployments in Prefect 3 - https://docs-3.prefect.io/v3/how-to-guides/deployment_infra/docker Option B: Keep the image local-only (no registry) - Make sure the worker uses the same Docker daemon where the image exists. - If your worker runs elsewhere, preload the image onto the worker: - Build on the worker host, or - docker save data_unload_local:local | ssh worker-host docker load - If your Docker work pool supports a pull policy, set it to IfNotPresent or Never so it won’t try to pull when the image exists locally. - Inspect your pool to see available variables: - prefect work-pool inspect docker-worker-pool - Look for keys like pull_policy or auto_pull in the base job template. If present, set via job_variables or pool template. If not present, preloading the image is the reliable route. Why your current config fails - Your call builds locally (default build=True) but push=False, so the worker’s host never gets the image. - The worker tries to pull data_unload_local:local from a registry; that’s not a published repo, so it’s denied. Minimal changes for each path - Registry path: - Change name to include a registry namespace and set push=True: image=DockerImage( name="ghcr.io/your-org/data_unload", tag=environment, dockerfile="data_flow.Dockerfile", rm=True ) - .deploy(..., push=True, ...) - Local-only path: - Keep your code as-is but ensure the worker has the image locally before the run: docker build -t data_unload_local:local -f data_flow.Dockerfile . # on the worker host docker load < image.tar # if you transfer it - Optionally set the pool’s pull policy (if available) to avoid pulling. If you tell me where your Docker worker runs relative to where you build this image (same host or remote), and whether you prefer using a registry or staying local-only, I’ll give you the exact config and commands to make it boringly reliable.
f
@Marvin docker worker is on the same host
m
thought for 109.4 seconds
Great — since the Docker worker is on the same host, this should be easy to make boringly reliable. What’s happening - Your deployment builds an image locally (build defaults to True), but push=False. - When the worker tries to start the flow, it can’t find that tag in its Docker daemon, so it attempts to pull. There’s no such repo, so it fails. - On the next run, the image may finally be present (or you hit the same daemon), so it works. Delightful. Two reliable fixes for a same-host Docker worker Option A: Pre-build once and skip building during deploy - Build the image on that host yourself: - docker build -t data_unload_local:local -f data_flow.Dockerfile . - Change your deploy call to use the existing local image and skip build/push: - .deploy(..., image=DockerImage(name=flow_name, tag=environment, dockerfile="data_flow.Dockerfile"), build=False, push=False, ...) - Verify the worker can see it: - docker images | grep data_unload_local - This avoids any chance of a pull. Option B: Keep deploy’s build, but make sure the worker sees the same Docker daemon - Ensure the deploy process and the worker use the same Docker context/daemon: - In both terminals (the one running deploy and the one running the worker), check: - docker context ls - echo $DOCKER_HOST (or check if you’re on Windows named pipe vs WSL vs remote context) - The error shows http+docker://localnpipe which is the Windows named pipe; ensure your build goes to the same daemon the worker uses. - Confirm the exact image name matches what’s built: - Your code uses name=flow_name and tag=environment; confirm the built image is exactly flow_name:environment - docker images | grep "${flow_name}" and look for the tag - If your Docker work pool’s base job template exposes a pull policy, set it to “IfNotPresent” (or “Never”) to stop any pull attempts when the image exists locally. You can inspect/update the pool template with: - prefect work-pool inspect docker-worker-pool - If you see a field for pull policy, set it accordingly and update the pool. If not exposed, pre-building (Option A) is the safest route. Small hardening tips - Use a stable, explicit image name to avoid accidental changes: - image=DockerImage(name="data_unload_local", tag=environment, dockerfile="data_flow.Dockerfile") - If you pre-build, always set build=False to avoid rebuilding and possible context mismatches. Reference - Docker deployments with Prefect 3: https://docs-3.prefect.io/v3/how-to-guides/deployment_infra/docker If you still see the pull attempts after pre-building, please share: - docker images | grep data_unload_local - docker context ls from both the deploy shell and the worker shell - The full image string your deployment resolves to (name:tag) We’ll make it stop pulling. It won’t be exciting, but it will stop crashing.