https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> Here's part of my docker compose: ``...
# marvin-ai
m
@Marvin Here's part of my docker compose:
Copy code
server:
    image: prefecthq/prefect:3.4.13-python3.11
    labels:
      - "com.azcare.service=work-wizard-server"
      - "com.azcare.domain=<http://demo-wizard.azcare.ai|demo-wizard.azcare.ai>"
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@db:5432/prefect>
      PREFECT_SERVER_API_HOST: 0.0.0.0
      PREFECT_UI_API_URL: "<https://demo-wizard.azcare.ai/api>"
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0
      PREFECT_LOGGING_LEVEL: DEBUG
      PREFECT_LOGGING_LOG_PRINTS: True
      PREFECT_SERVER_LOGGING_LEVEL: DEBUG
      PREFECT_SERVER_LOGGING_LOG_PRINTS: True
      PREFECT_SERVER_API_AUTH_STRING: "admin:password"
    command: prefect server start --no-services
    ports:
      - "4200:4200"
    healthcheck:
      test: [ "CMD-SHELL", "python -c 'import json,urllib.request,sys; sys.exit(0 if json.loads(urllib.request.urlopen(\"<http://localhost:4200/api/health>\", timeout=1).read()) is True else 1)' " ]
      interval: 5s
      timeout: 3s
      retries: 30
    restart: always
create-autoupdate-deployment:
    image: prefect-workpool
    build:
      context: .
      dockerfile: docker/autoupdate/Dockerfile
    depends_on:
      server:
        condition: service_healthy
    environment:
      PREFECT_API_URL: <http://server:4200/api>
      PREFECT_SERVER_API_AUTH_STRING: "admin:password"
    # command: prefect deployment build autoupdate/pipeline/deployment.py:autoupdate_pipeline --name autoupdate-pipeline --cron "0 0 * * *" --timezone "America/New_York" --infra prefect-wizard-infra --storage-block prefect-s3-block
    command: bash -c "cd /opt/prefect/autoupdate && prefect --no-prompt deploy --name autoupdate-deployment"
    volumes:
      - ./autoupdate:/opt/prefect/autoupdate
      - ./core:/opt/prefect/autoupdate/core
    env_file:
      - .env

  autoupdate-worker:
    image: prefect-workpool
    depends_on:
      server:
        condition: service_healthy
      services:
        condition: service_started
      create-autoupdate-deployment:
        condition: service_completed_successfully
    environment:
      PREFECT_API_URL: <http://localhost:4200/api>
      DOCKER_HOST: <tcp://localhost:2375>
      PREFECT_SERVER_API_AUTH_STRING: "admin:password"
      # DOCKER_TLS_VERIFY: ${DOCKER_TLS_VERIFY:-}
      # DOCKER_CERT_PATH: ${DOCKER_CERT_PATH:-}
    network_mode: host
    volumes:
      - ./autoupdate:/opt/prefect/autoupdate
      - ./core:/opt/prefect/autoupdate/core
      - ${HOME}/.wizard:/logs
    env_file:
      - .env
    restart: always
I get this auth error with create-autoupdate-deployment:
Copy code
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 379, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 163, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url '<http://server:4200/api/work_pools/auto-update-pool>'
Response: {'exception_message': 'Unauthorized'}
For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401>
m
✅ thought for 97.3 seconds
You’re hitting 401 because the client containers aren’t sending credentials. In Prefect 3.x: - Server-side: set PREFECT_SERVER_API_AUTH_STRING to enable Basic Auth on the API. - Client-side (CLI, workers, agents, etc.): set PREFECT_API_AUTH_STRING so the client library actually sends the Authorization header on every request. - PREFECT_API_KEY (Cloud) takes precedence over PREFECT_API_AUTH_STRING. If it’s present in your .env, your client will send a bearer token instead of Basic Auth and your self-hosted server will reject it. What to change in your compose: - Keep PREFECT_SERVER_API_AUTH_STRING on the server. - Replace PREFECT_SERVER_API_AUTH_STRING with PREFECT_API_AUTH_STRING in create-autoupdate-deployment and autoupdate-worker. - Ensure PREFECT_API_KEY is not set (or override it to empty) in those client services. Example adjustments:
Copy code
server:
  environment:
    PREFECT_SERVER_API_AUTH_STRING: "admin:password"
    PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@db:5432/prefect>
    PREFECT_SERVER_API_HOST: 0.0.0.0
    PREFECT_UI_API_URL: "<https://demo-wizard.azcare.ai/api>"
    # ...other settings...

create-autoupdate-deployment:
  environment:
    PREFECT_API_URL: <http://server:4200/api>
    PREFECT_API_AUTH_STRING: "admin:password"
    PREFECT_API_KEY: ""  # ensure not set from .env
  # command: make sure you have '&&' not '&&' in the actual file
  command: bash -c "cd /opt/prefect/autoupdate && prefect --no-prompt deploy --name autoupdate-deployment"

autoupdate-worker:
  environment:
    PREFECT_API_URL: <http://localhost:4200/api>
    PREFECT_API_AUTH_STRING: "admin:password"
    PREFECT_API_KEY: ""  # ensure not set from .env
    DOCKER_HOST: <tcp://localhost:2375>
  # ...rest unchanged...
Docs for reference: - Security settings (Prefect 3) If you still see 401s after this change, please double-check: - No PREFECT_API_KEY is leaking in via your .env. - The password matches exactly between server and clients. - You can curl the API with basic auth from the client container, e.g. 
curl -u admin:password <http://server:4200/api/health>
(should return true).
m
@Marvin That didn't work Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/prefect/cli/_utilities.py", line 44, in wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/cli/_types.py", line 156, in sync_fn return asyncio.run(async_fn(*args, **kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/cli/deploy.py", line 483, in deploy await _run_single_deploy( File "/usr/local/lib/python3.11/site-packages/prefect/cli/deploy.py", line 548, in _run_single_deploy work_pool = await client.read_work_pool(deploy_config["work_pool"]["name"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration/_work_pools/client.py", line 416, in read_work_pool response = await self.request( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration/base.py", line 53, in request return await self._client.send(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 379, in send response.raise_for_status() File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 163, in raise_for_status raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.cause prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url 'http://server:4200/api/work_pools/auto-update-pool' Response: {'exception_message': 'Unauthorized'} For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401 An exception occurred.
m
✅ thought for 73.9 seconds
Thanks for the details — let’s zero in on why the client still isn’t sending creds. The 401 on /api/work_pools/... means the server is enforcing Basic auth, but the client request is missing or has the wrong Authorization header. The most common causes are: - The client image is running a Prefect version that doesn’t support/recognize PREFECT_API_AUTH_STRING (e.g., older than 3.x) - PREFECT_API_AUTH_STRING isn’t actually set at runtime (or is being shadowed by PREFECT_API_KEY from .env) - The client is pointing at the wrong API URL Can you run these quick checks inside the create-autoupdate-deployment container and share the output? 1) Verify Prefect version and that settings are applied:
Copy code
prefect version
prefect config view | egrep -i 'api\.(url|auth_string|key)'
python - <<'PY'
from prefect.settings import get_current_settings
s = get_current_settings()
print("api.url =", s.api_url)
print("api.auth_string =", s.api_auth_string)
print("api.key =", s.api_key)
PY
2) Verify the API requires Basic auth and that your creds work:
Copy code
curl -sv <http://server:4200/api/health>
curl -sv -u admin:password <http://server:4200/api/health>
- First call should 401 - Second should 200 with “true” 3) Print env to ensure nothing is overriding:
Copy code
env | grep -E '^PREFECT_'
Confirm: - PREFECT_API_URL=http://server:4200/api - PREFECT_API_AUTH_STRING=admin:password - PREFECT_API_KEY is empty/unset - No stray PREFECT_CLOUD_* variables If any of the above don’t match, that’s the smoking gun. A couple of fixes I recommend right away: - Ensure your client image is on the same Prefect version as the server (you’re running server 3.4.13). If your custom prefect-workpool image doesn’t pin Prefect, it could be older/newer and ignore the setting. Easiest: base it on the official image or pip-pin it. - Example Dockerfile base: FROM prefecthq/prefect:3.4.13-python3.11 - Or ensure: pip install "prefect==3.4.13" - In the create-autoupdate-deployment service, explicitly neutralize any API key from .env:
Copy code
environment:
  PREFECT_API_URL: <http://server:4200/api>
  PREFECT_API_AUTH_STRING: "admin:password"
  PREFECT_API_KEY: ""   # make sure this wins over .env
Optional sanity check: run the deploy once with inline env to rule out compose/env_file issues:
Copy code
bash -lc 'PREFECT_API_URL=<http://server:4200/api> PREFECT_API_AUTH_STRING=admin:password PREFECT_API_KEY= prefect --no-prompt deploy --name autoupdate-deployment'
If the above checks still show 401, please share: - Output of the four commands above - The Dockerfile for prefect-workpool - A short snippet of the server logs around the 401 (since you have PREFECT_SERVER_LOGGING_LEVEL=DEBUG) Reference: - Security settings (Prefect 3) (PREFECT_SERVER_API_AUTH_STRING on the server, PREFECT_API_AUTH_STRING on clients)
m
ow to ensure work pool is ready to receive jobs
@Marvin how to ensure work pool is ready to receive jobs
m
✅ thought for 181.7 seconds
Great question. In Prefect 3.x, a work pool is “ready to receive jobs” when: - The pool exists and is not paused - At least one worker is running and polling that pool (and queue, if specified) - The pool’s base job template is valid for the worker type (e.g., docker worker for a docker pool) - Concurrency limit is not 0 - Your deployments are assigned to that pool (and queue, if using queues) Here’s a concise readiness checklist with verified CLI commands: 1) Confirm the pool exists and is not paused - Create (idempotent) and resume:
Copy code
prefect work-pool create "auto-update-pool" --type docker --overwrite
prefect work-pool resume "auto-update-pool"
- Inspect details:
Copy code
prefect work-pool inspect "auto-update-pool"
2) Ensure a worker is running and polling the pool - Start a worker:
Copy code
prefect worker start -p "auto-update-pool"
- Check worker logs for lines like “Connected to work pool 'auto-update-pool'” and “Polling for work”. 3) Validate concurrency and pausing - Make sure the pool is not paused (see inspect output). - If you set a concurrency limit, make sure it’s > 0:
Copy code
prefect work-pool set-concurrency-limit "auto-update-pool" 10
4) Confirm your deployment points at the pool (and queue, if any) - Inspect the deployment:
Copy code
prefect deployments inspect "<your-deployment-name>"
- Ensure it lists Work pool: auto-update-pool (and the queue name if used). If not, update and redeploy. 5) Run a quick smoke test - Trigger a run:
Copy code
prefect deployments run "<your-deployment-name>"
- You should see the worker pick it up. In your compose specifically - Make sure your worker service actually starts a worker. Your snippet didn’t show a command; add:
Copy code
command: >
  bash -lc 'prefect worker start -p "auto-update-pool"'
- Ensure the pool exists before the worker starts. Add a small init service:
Copy code
create-work-pool:
  image: prefecthq/prefect:3.4.13-python3.11
  depends_on:
    server:
      condition: service_healthy
  environment:
    PREFECT_API_URL: <http://server:4200/api>
    PREFECT_API_AUTH_STRING: "admin:password"
  command: >
    bash -lc 'prefect work-pool create "auto-update-pool" --type docker --overwrite &&
              prefect work-pool resume "auto-update-pool" || true'
- Then depend your worker on this:
Copy code
autoupdate-worker:
  depends_on:
    server:
      condition: service_healthy
    create-work-pool:
      condition: service_completed_successfully
  command: >
    bash -lc 'prefect worker start -p "auto-update-pool"'
  environment:
    PREFECT_API_URL: <http://localhost:4200/api>
    PREFECT_API_AUTH_STRING: "admin:password"
    PREFECT_API_KEY: ""
Optional: a simple readiness gate you can run before deployments
Copy code
# assumes PREFECT_API_URL and PREFECT_API_AUTH_STRING are set
prefect work-pool inspect "auto-update-pool" >/dev/null 2>&1 || exit 1
# If you pause pools in automation, also parse the output to verify not paused
Notes for your environment - You’re using Docker: ensure your pool type is docker and the worker can reach the Docker daemon (DOCKER_HOST etc.). - With Basic Auth: set PREFECT_API_URL and PREFECT_API_AUTH_STRING on the worker and deployment containers; ensure PREFECT_API_KEY is unset/empty so Basic Auth is used. If you want, share: - The output of 
prefect work-pool inspect "auto-update-pool"
- The worker container logs after adding the start command Docs: - Work pools and workers CLI commands are available via 
prefect work-pool --help
and 
prefect worker --help
.
b
@Mohamed Zaki If you're using Redis for HA/multi-server you'll need to add 
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
to your environment variables to ensure correct ordering of events. The docs got updated on Saturday so you may have not seen this. https://github.com/PrefectHQ/prefect/issues/18766#issuecomment-3215942971
Did you resolve the 401 error with your worker?
Open in Slack
PreviousNext
Powered by