docker.errors.ImageNotFound: 404 Client Error for ...
# ask-marvin
p
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.52/images/create?tag=latest&fromImage=google-maps-scraper: Not Found ("pull access denied for google-maps-scraper, repository does not exist or may require 'docker login'") 095908 AM prefect.flow_runs.worker Reported flow run '7f17c8b3-b47a-4000-9d1b-f7ed3b11286e' as crashed: Flow run could not be submitted to infrastructure: ImageNotFound(HTTPError('404 Client Error: Not Found for url: http+docker://localhost/v1.52/images/create?tag=latest&fromImage=google-maps-scraper')) from pathlib import Path from loguru import logger from prefect import flow from prefect.docker import DockerImage if name == "__main__": root_path = Path(file).parent.parent logger.info(f"Deploying from root path: {root_path}") flow_to_deploy = flow.from_source( source=root_path, entrypoint="workflows/main.py:main_flow", ) docker_file_path = root_path / "Dockerfile" logger.info(f"Using Dockerfile at: {docker_file_path}") print(f"Flow to deploy: {flow_to_deploy}") if hasattr(flow_to_deploy, "deploy"): flow_to_deploy.deploy( # pyright: ignore[reportAttributeAccessIssue] "Google Maps Scraper Deployment", work_pool_name="docker-work-pool", image=DockerImage( name="google-maps-scraper", tag="latest", dockerfile=str(docker_file_path) ), push=False, ) why it raise that error @Marvin?
m
thought for 285.1 seconds
Short answer: your Docker worker is trying to pull the image
google-maps-scraper:latest
from a registry (Docker Hub by default), but it isn’t there (and you set
push=False
), so it fails with 404/pull access denied. With the
latest
tag, the Docker worker’s default pull policy is Always, so it will try to pull even if the image exists locally. Why this happens in Prefect 3.x: -
flow.deploy(..., image=DockerImage(...), push=False)
builds the image locally during deployment but does not push it to any registry. - The Docker worker resolves image pull policy like Kubernetes: - Tag
latest
or no tag => Always pull - Specific tag (not
latest
) => IfNotPresent by default - Because your image isn’t pushed anywhere, the worker’s pull attempt fails with “repository does not exist or may require 'docker login'”. Fix options (pick the one that matches your setup): 1) Push to a registry and let the worker pull - Use a fully qualified image name and push=True. - If it’s a private registry, configure registry credentials on the work pool.
Copy code
flow_to_deploy.deploy(
    "Google Maps Scraper Deployment",
    work_pool_name="docker-work-pool",
    image=DockerImage(
        name="<http://docker.io/<your-dockerhub-username>/google-maps-scraper|docker.io/<your-dockerhub-username>/google-maps-scraper>",
        tag="0.1.0",
        dockerfile=str(docker_file_path),
    ),
    push=True,  # push so the worker can pull
)
- If private, add credentials on the work pool (UI or job variables) using a DockerRegistryCredentials block. 2) Keep everything local on the same machine as the worker (no registry) - Tag with something other than latest and set pull policy to Never so the worker uses the local image only.
Copy code
flow_to_deploy.deploy(
    "Google Maps Scraper Deployment",
    work_pool_name="docker-work-pool",
    image=DockerImage(
        name="google-maps-scraper",
        tag="0.1.0",
        dockerfile=str(docker_file_path),
    ),
    push=False,
    job_variables={"image_pull_policy": "Never"},
)
Important: This only works if the worker is running on the same host and the image exists locally on that host. 3) Use a non-latest tag and rely on IfNotPresent - If the image is already present locally on the worker host:
Copy code
image=DockerImage(name="google-maps-scraper", tag="0.1.0", dockerfile=str(docker_file_path))
push=False
# With a non-latest tag, default policy is IfNotPresent; the worker will use local if available.
If the worker doesn’t have the image, it will still try to pull and fail — in that case, either push to a registry (option 1) or set Never and ensure the image is built on the worker host (option 2). Helpful docs: - Docker work pools, image_pull_policy, job variables: https://docs.prefect.io/v3/how-to-guides/deployment_infra/docker/ - Deploy via Python with DockerImage: https://docs.prefect.io/v3/how-to-guides/deployments/deploy-via-python/ - Customize job variables (including pull policy and registry credentials): https://docs.prefect.io/v3/how-to-guides/deployments/customize-job-variables/ Questions to confirm next steps: - Is your Docker worker running on the same machine where you call deploy? - Do you want to avoid using a registry, or should we push to Docker Hub/GHCR/ECR and configure credentials?
p
how to setup docker push ghcr @Marvin?
m
thought for 119.9 seconds
Got it — you want to push your Prefect image to GitHub Container Registry (GHCR) and have your Docker worker pull it. There are two parts: 1) Push the image to GHCR 2) Configure your Prefect Docker work pool to authenticate to GHCR when pulling 1) Push your image to GHCR - Create a GitHub Personal Access Token (classic) with scope: read:packages (pull) and write:packages (push) - Login to GHCR locally
Copy code
echo YOUR_GHCR_TOKEN | docker login <http://ghcr.io|ghcr.io> -u YOUR_GITHUB_USERNAME --password-stdin
- Tag your image with the GHCR path and push
Copy code
# Example names
ORG=your-gh-org-or-username
IMAGE=google-maps-scraper
TAG=0.1.0

docker tag ${IMAGE}:latest <http://ghcr.io/${ORG}/${IMAGE}:${TAG}|ghcr.io/${ORG}/${IMAGE}:${TAG}>
docker push <http://ghcr.io/${ORG}/${IMAGE}:${TAG}|ghcr.io/${ORG}/${IMAGE}:${TAG}>
Tip: Prefer a specific tag (not latest) for reproducibility. If you’re using Prefect’s deploy build step: - Set the full image name and push=True so Prefect builds and pushes it for you.
Copy code
flow_to_deploy.deploy(
    "Google Maps Scraper Deployment",
    work_pool_name="docker-work-pool",
    image=DockerImage(
        name=f"<http://ghcr.io/{ORG}/{IMAGE}|ghcr.io/{ORG}/{IMAGE}>",
        tag=TAG,
        dockerfile=str(docker_file_path),
    ),
    push=True,
)
2) Configure your Prefect Docker work pool to pull from GHCR Create a Docker Registry Credentials block for GHCR, then reference it from your deployment or work pool. A) Create the credentials block (one-time) - UI: Blocks -> + -> Docker Registry Credentials - Username: your GitHub username - Password: your GHCR PAT - Registry URL: ghcr.io - Save as: ghcr-credentials - Or via Python:
Copy code
from prefect_docker import DockerRegistryCredentials

DockerRegistryCredentials(
    username="YOUR_GITHUB_USERNAME",
    password="ghp_xxxxxxxx",   # PAT with read:packages (and write:packages if pushing)
    registry_url="<http://ghcr.io|ghcr.io>",
    reauth=True,
).save("ghcr-credentials", overwrite=True)
B) Reference the block so the worker logs in before pulling - Easiest: set in your deploy call’s job_variables
Copy code
flow_to_deploy.deploy(
    "Google Maps Scraper Deployment",
    work_pool_name="docker-work-pool",
    image=f"<http://ghcr.io/{ORG}/{IMAGE}:{TAG}|ghcr.io/{ORG}/{IMAGE}:{TAG}>",
    push=True,  # or False if you already pushed manually
    job_variables={
        "registry_credentials": "{{ prefect.blocks.docker-registry-credentials.ghcr-credentials }}",
        # Optional: control pull policy
        "image_pull_policy": "IfNotPresent"  # or "Always" / "Never"
    },
)
- Or set it at the work pool level (applies to all deployments): - Work Pools -> your docker pool -> Edit -> set Image (optional) and Registry Credentials to the same block Key notes: - With tag latest, Docker worker defaults to Always pull. If you won’t push yet, use a non-latest tag and optionally set image_pull_policy to Never for local-only development. - Make sure the image name in your deployment exactly matches what you pushed, e.g. ghcr.io/org/google-maps-scraper:0.1.0 If you share your org/repo layout and whether you want Prefect to build/push during deploy or you’ll push separately, I can give you a tailored one-command setup.