It sounds like the flow-run container isn’t getting Prefect auth/env vars. In Prefect 3.x, the Docker worker’s process can be authenticated to poll your work pool, but containers it launches do NOT automatically inherit those credentials. Each flow-run container must have its own

PREFECT_API_URL

and either

PREFECT_API_KEY

(Cloud or API-key-backed server) or

PREFECT_API_AUTH_STRING

(basic auth on self-hosted) set. Why you’re seeing 401 inside the container - The worker can talk to the API, but the flow-run container likely has no

PREFECT_API_URL

and/or no valid

PREFECT_API_KEY

(or it’s pointing at an unreachable URL like localhost). - If you’re self-hosting with basic auth and you set

PREFECT_API_KEY

by mistake, the client will try key auth and fail. Don’t set both; use only the one that matches your server’s auth method. How to fix (pick one place to set env for the container) 1) At the Work Pool (applies to all deployments on that pool) - In the UI: Work Pools -> your Docker pool -> Edit -> Default job variables -> env - Add: - `PREFECT_API_URL`: your workspace or server URL (Cloud example below) - `PREFECT_API_KEY`: your service account key (for Cloud), OR - `PREFECT_API_AUTH_STRING`:

username:password

(for self-hosted basic auth) - CLI (inspect the template to see the env hook):

prefect work-pool inspect "my-docker-pool"

You’ll see

env: "{{ env }}"

in the job template; set Default job variables -> env in the UI or via deployment (below). 2) In your deployment (only for that deployment) - In your

prefect.yaml

for the deployment:

deployments:
    - name: hello-docker
      entrypoint: flows/hello.py:hello_flow
      work_pool:
        name: my-docker-pool
      job_variables:
        env:
          PREFECT_API_URL: <https://api.prefect.cloud/api/accounts/<account_id>/workspaces/<workspace_id>>
          PREFECT_API_KEY: ${PREFECT_API_KEY}

Then run:

prefect deploy

Tip: Provide

PREFECT_API_KEY

via your shell/CI env so it’s not hardcoded. 3) Bake into the image (least recommended for secrets) - Add

ENV PREFECT_API_URL=...

and

ENV PREFECT_API_KEY=...

to your Dockerfile (or better, pass them at runtime), but avoid hardcoding secrets in images. Cloud vs. self-hosted specifics - Prefect Cloud: -

PREFECT_API_URL

must point to your Cloud workspace URL:

<https://api.prefect.cloud/api/accounts/<account_id>/workspaces/<workspace_id>>

- Use a service account API key (recommended for workers/containers): create one here: Service accounts docs - Self-hosted: - Ensure the URL is reachable from inside the container. If your server is on the same host, do not use

<http://localhost:4200>

inside the container. Use the host’s network address or

host.docker.internal

(on Docker Desktop) or put both on a Docker network. - If you enabled basic auth, set only

PREFECT_API_AUTH_STRING=username:password

. Do not set

PREFECT_API_KEY

. Quick diagnostics - Check what the container actually sees:

printenv | grep PREFECT_

You should see

PREFECT_API_URL

and either

PREFECT_API_KEY

or

PREFECT_API_AUTH_STRING

. - Confirm your pool’s base job template has the env hook:

prefect work-pool get-default-base-job-template --type docker

You should see

"env": "{{ env }}"

in the job configuration, meaning whatever you put in job_variables.env is passed to the container. Helpful docs - Connect to Cloud - Service accounts (Cloud) - Customize job variables - Manage work pools If you can share: - Whether you’re on Prefect Cloud or self-hosted

- Your Docker work pool’s Default job variables (env) snippet (redact secrets) - The

job_variables

section of your

prefect.yaml

(if you’re setting it there) - The exact

PREFECT_API_URL

you’re using …I can pinpoint the exact tweak to make.

Thanks for the detail — that helps. The short answer: the Prefect client inside your flow-run container does not read

PREFECT_SERVER_API_AUTH_STRING

. In Prefect 3.x, the client only respects: -

PREFECT_API_URL

-

PREFECT_API_KEY

(for Prefect Cloud or any API-key-based server) -

PREFECT_API_AUTH_STRING

(for self‑hosted server with basic auth in the form

username:password

) If more than one auth mechanism is present, the API key takes precedence. Variables with “SERVER” in the name are not used by the client. Having both auth styles present (or an incorrect one for your target) commonly results in 401s. What likely happened - If you’re targeting Prefect Cloud but only set

PREFECT_API_AUTH_STRING

(or also set

PREFECT_SERVER_API_AUTH_STRING

), the client will ignore it and look for

PREFECT_API_KEY

— missing or wrong -> 401. - If you’re self-hosted with basic auth and both

PREFECT_API_KEY

and

PREFECT_API_AUTH_STRING

are set anywhere (even empty key), the key wins and you’ll get 401 if it’s invalid. - Occasionally, injected secrets add a trailing newline or space to

PREFECT_API_AUTH_STRING

so the

username:password

is not exactly correct. Concrete fixes (pick the one that matches your setup) - Prefect Cloud: - Ensure

PREFECT_API_URL

points to your workspace URL (e.g. `https://api.prefect.cloud/api/accounts/<account_id>/workspaces/<workspace_id>`) - Set only

PREFECT_API_KEY

(ideally a service account key). Remove both

PREFECT_API_AUTH_STRING

and

PREFECT_SERVER_API_AUTH_STRING

from the container. - Self-hosted with basic auth: - Ensure

PREFECT_API_URL

points to your API (e.g. `http://host:4200/api`) - Set only

PREFECT_API_AUTH_STRING=USERNAME:PASSWORD

- Make sure

PREFECT_API_KEY

is completely unset in the container. Remove

PREFECT_SERVER_API_AUTH_STRING

(it’s not used by the client). Quick checks you can run inside the failing container

# See exactly what’s set
printenv | grep PREFECT_

# Sanity-check your values
python - <<'PY'
import os, asyncio
from prefect.client.orchestration import get_client
print("URL:", os.getenv("PREFECT_API_URL"))
print("API_KEY set?:", bool(os.getenv("PREFECT_API_KEY")))
print("AUTH_STRING set?:", bool(os.getenv("PREFECT_API_AUTH_STRING")))
print("AUTH_STRING repr:", repr(os.getenv("PREFECT_API_AUTH_STRING")))
async def main():
    async with get_client() as c:
        await c.api_healthcheck()
        print("Client ok")
asyncio.run(main())
PY

If that still returns 401: - For Cloud, confirm the key format (

pnu_

or

pnb_

) and that the key has access to the target workspace. - For self-hosted basic auth, verify the exact credentials your API expects and check for trailing whitespace in

PREFECT_API_AUTH_STRING

. Helpful docs - Connect to Cloud - Service accounts (Cloud) - Customize job variables - Manage work pools If you can share: - Whether you’re on Cloud or self-hosted - The

PREFECT_API_URL

you’re using - Whether

PREFECT_API_KEY

is set anywhere in the container …I can give you the exact minimal env to keep and which ones to remove.

Got it — thanks for clarifying you’re on self-hosted. Here’s what’s going on and how to fix it: Key point - The Prefect client only reads these auth settings: -

PREFECT_API_URL

-

PREFECT_API_AUTH_STRING

(basic auth, format:

username:password

) -

PREFECT_API_KEY

(not used for self-hosted basic auth) -

PREFECT_SERVER_API_AUTH_STRING

is not read by the client. Workers sometimes forward both for compatibility, but the client ignores the SERVER one. Why you can still get 401 even if both strings look “correct” - If

PREFECT_API_KEY

is present anywhere in the container env (even empty), it takes precedence over

PREFECT_API_AUTH_STRING

and will cause 401 against a self-hosted server. This is the most common surprise. - Trailing whitespace/newlines or quotes in

PREFECT_API_AUTH_STRING

can make the Basic header invalid. - If your API is behind a reverse proxy with a path prefix (e.g.

/prefect/api

),

PREFECT_API_URL

must include that exact base; otherwise the proxy may return 401 from a different auth layer. - If the username/password you expect are being set through a secret manager, they sometimes include a newline character. Concrete checks inside the failing container 1) Look for any API key overriding your basic auth:

printenv | grep PREFECT_API_

You should NOT see

PREFECT_API_KEY

at all. 2) Check exact values and whitespace:

python - <<'PY'
import os, reprlib
print("URL:", os.getenv("PREFECT_API_URL"))
print("API_KEY present?:", "PREFECT_API_KEY" in os.environ)
val = os.getenv("PREFECT_API_AUTH_STRING")
print("AUTH_STRING present?:", val is not None)
print("AUTH_STRING repr:", repr(val))
PY

3) Sanity test the same creds with curl:

# Extract values from env first
# Substitute values from the previous print if needed
curl -v -u 'username:password' "$PREFECT_API_URL/flow_runs/<your_flow_run_id>"

You should see a 200 or JSON response. If curl works but the client still 401s, it’s almost always because

PREFECT_API_KEY

is present and winning precedence. 4) Programmatically verify the Prefect client auth:

python - <<'PY'
import asyncio
from prefect.client.orchestration import get_client
async def main():
    async with get_client() as c:
        await c.api_healthcheck()
        print("Client authenticated successfully")
asyncio.run(main())
PY

What to change - Keep:

PREFECT_API_URL

,

PREFECT_API_AUTH_STRING

- Remove entirely:

PREFECT_API_KEY

(if present anywhere),

PREFECT_SERVER_API_AUTH_STRING

(harmless but unused) - Ensure

PREFECT_API_URL

includes any proxy path prefix and ends at your API base (e.g.

<http://your-host:4200/api>

or `https://your-proxy/prefect/api`) - Make sure

PREFECT_API_AUTH_STRING

has no quotes or trailing whitespace: - Good:

PREFECT_API_AUTH_STRING=user:pass

- Bad:

PREFECT_API_AUTH_STRING='user:pass'

or

user:pass\n

If you share: - The exact

PREFECT_API_URL

value - The output of

printenv | grep PREFECT_API_

inside the flow container (you can redact the password) - Whether a reverse proxy is in front of the API (and its path prefix) …I can pinpoint the exact env var to remove or adjust. Reference docs: - Security settings (self-hosted) - Customize job variables - Manage work pools