That error is coming from Docker, not Prefect. Your worker is asking the local Docker daemon to pull from Google Artifact Registry (GAR), and Docker doesn’t have credentials. Granting a GCE service account permission is necessary, but Docker still needs a way to present those credentials. What you need to do: configure Docker auth on the machine that runs the worker and Docker engine. Prefect will happily use whatever Docker can use; it does not inject registry credentials for you. Recommended setup (GAR) - Ensure the service account that should pull images has roles/artifactregistry.reader on the repo or project. - On the worker host, install the Google Cloud SDK, then configure the Docker credential helper for your GAR hostname:

gcloud auth configure-docker us-central1-docker.pkg.dev

This writes a credHelpers entry in your Docker config so Docker can fetch/refresh tokens automatically via gcloud. - Verify from the same environment that your Prefect worker will use:

docker pull us-central1-docker.pkg.dev/PROJECT/REPOSITORY/IMAGE:TAG

If this fails, Prefect will fail too. Alternatives (useful for non-interactive or locked-down hosts) - Short-lived token (simple, but expires ~1 hour):

gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin <https://us-central1-docker.pkg.dev>

- Service account key (persists until key rotation; handle with care):

docker login -u _json_key -p "$(cat /path/to/key.json)" <https://us-central1-docker.pkg.dev>

Common gotchas - Wrong user/HOME: Docker reads credentials from

$DOCKER_CONFIG/config.json

or

$HOME/.docker/config.json

. If you run the Prefect worker as a different user (systemd, container, sudo), it may not see your credentials. - Either place

config.json

under that user’s home, or set

DOCKER_CONFIG

for the worker process. - Example systemd override:

[Service]
    Environment=DOCKER_CONFIG=/home/prefect/.docker

- Hostname mismatch: make sure the registry you log into matches exactly what you pull, e.g.

us-central1-docker.pkg.dev

. - Test outside Prefect first: run

docker pull

from the same shell/service environment that runs

prefect worker start

. If that fails, fix auth there before blaming the poor worker. Quick checklist to fix your case 1. On the VM running the worker: -

gcloud auth configure-docker us-central1-docker.pkg.dev

- Confirm the active identity has access (or that gcloud is using the VM’s service account). 2. Ensure the Prefect worker process sees the same Docker config: - If you start via CLI in a shell, it usually does. - If it’s a service, set

DOCKER_CONFIG

as above. 3. Test:

docker pull us-central1-docker.pkg.dev/mlops-461404/xxx/IMAGE:TAG

4. Restart the worker and try again:

prefect worker start -p <your-docker-work-pool>

If you’re on Prefect 2.x with a Docker work pool, the above still applies; Prefect 3.x primarily uses the Process worker, but either way, Docker auth is handled by the host. Helpful docs - Google Artifact Registry auth for Docker: GAR authentication docs - Prefect 3 concepts: workers and execution (background on how workers interact with infra) Prefect 3: Workers - Serverless/infra overview (for context) Prefect 3: Serverless/infra If you can share: - Prefect version (2.x Docker work pool vs 3.x) - How you’re starting the worker (CLI vs systemd/container) and which user it runs as - Whether

docker pull ...

succeeds from that same environment …I can point you to the exact incantation you need. I know, thrilling.

Thanks for the compose — that explains the “why.” You’re running the Prefect worker inside a container and mounting the Docker socket. That means: - The Docker “client” lives inside the worker container. - The client inside the container must have registry credentials for us-central1-docker.pkg.dev. - Credentials on the host (gcloud on the VM or the VM’s service account) aren’t visible to the client in the container unless you explicitly pass them in. Pick one of these fixes. Option A — Mount a Docker config that already contains GAR creds (no gcloud helper) 1) On the host, create creds using a JSON key so Docker writes plain auth to config.json (not a gcloud credHelper):

docker login -u _json_key -p "$(cat /path/to/gar-key.json)" <https://us-central1-docker.pkg.dev>

2) Mount that config into the worker container and use the same DOCKER_HOST:

version: "3.9"

services:
  prefect-worker:
    image: prefecthq/prefect:3-latest
    container_name: prefect-worker-pool
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /root/.docker:/root/.docker:ro  # adjust to wherever you ran `docker login`
    environment:
      - PREFECT_API_URL=<http://xxx:4200/api>
      - PREFECT_API_AUTH_STRING=admin:xxx
      - PREFECT_SERVER_API_AUTH_STRING=admin:xxx
      - DOCKER_HOST=unix:///var/run/docker.sock
    command: ["prefect", "worker", "start", "--install-policy", "always", "--pool", "my-docker-pool"]

Important: Do NOT use

gcloud auth configure-docker

for this mount approach unless you also install the gcloud credential helper inside the container. The helper referenced in config.json must exist where the client runs (inside the container), not just on the host. Option B — Login inside the container at startup using a Docker secret If you don’t want to mount your host’s `~/.docker`:

version: "3.9"

services:
  prefect-worker:
    image: prefecthq/prefect:3-latest
    container_name: prefect-worker-pool
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:rw
    secrets:
      - gar_key
    environment:
      - PREFECT_API_URL=<http://xxx:4200/api>
      - PREFECT_API_AUTH_STRING=admin:xxx
      - PREFECT_SERVER_API_AUTH_STRING=admin:xxx
      - DOCKER_HOST=unix:///var/run/docker.sock
    entrypoint: ["/bin/sh","-lc"]
    command: >
      docker login -u _json_key -p "$(cat /run/secrets/gar_key)" <https://us-central1-docker.pkg.dev> &&
      prefect worker start --install-policy always --pool my-docker-pool

secrets:
  gar_key:
    file: ./gar-key.json

Option C — Use gcloud inside the container (only if you really want ADC/cred helpers) Build a small image that installs gcloud and run

gcloud auth configure-docker us-central1-docker.pkg.dev

in the container. This works, but it’s heavier than A/B. Sanity checks - Remove angle brackets around the API URL in compose: use

<http://xxx:4200/api>

, not

<http://...>

. - From inside the running worker container, verify:

docker pull us-central1-docker.pkg.dev/mlops-461404/xxx/IMAGE:TAG

If this fails, Prefect will fail too. - Make sure the service account you’re using has at least roles/artifactregistry.reader on the repo/project. - The registry host must match exactly:

us-central1-docker.pkg.dev

. Docs, in case you like bedtime reading - GAR auth for Docker: Google Artifact Registry authentication - Prefect 3 workers overview: Prefect 3: Workers If you tell me whether you prefer mounting a config (A) or logging in at start with a secret (B), I’ll hand you a ready-to-paste compose specific to your paths. I live to serve. Sort of.