This message was deleted.
# ask-marvin
s
This message was deleted.
m
thought for 167.3 seconds
Thanks for the log — a 503 from the API means the Prefect server is up but not able to serve the request (typically due to database/readiness or a proxy issue). One thing jumps out in your URL: - I see
%27
(a URL-encoded single quote) at the end:
.../api/task_runs/%27
- That strongly suggests your
PREFECT_API_URL
likely has a stray trailing quote, e.g.
<http://prefect-server:4200/api>'
- If so, fix that to exactly
<http://prefect-server:4200/api>
(no quotes), then restart the worker/flow environment Beyond that, 503s usually come from: - Server not ready yet (migrations still running) or database not reachable - Database locks/exhausted pool (SQLite or undersized Postgres) - Load balancer/proxy returning 503 for an unhealthy upstream - DNS/network issue between your worker and the server hostname Quick checks 1) Verify your API URL and connectivity from the worker
Copy code
prefect config view | grep PREFECT_API_URL
echo "$PREFECT_API_URL"
curl -v "$PREFECT_API_URL/health"
curl -v "$PREFECT_API_URL/ready"
- /api/health should return 200 true - /api/ready should return 200 OK; if 503, the server is not ready (often DB-related) 2) Check server logs for DB/readiness/migrations - Look for “Database is not available”, “migration”, “alembic”, “OperationalError”, “SQLITE_BUSY” - If using Docker Compose:
Copy code
docker logs <prefect-server-container> --since=30m
- If using Kubernetes:
Copy code
kubectl logs deploy/prefect-server -n <ns> --since=30m
3) Check database connectivity (if self-hosted DB) - From the server pod/container, validate DB reachability:
Copy code
psql -h <db-host> -U <db-user> -d <db-name> -c "SELECT 1"
# or MySQL:
mysql -h <db-host> -u <db-user> -p<pass> -e "SELECT 1"
- Ensure
PREFECT_API_DATABASE_CONNECTION_URL
is correct and the DB accepts connections - If using SQLite and seeing lock errors, consider Postgres for concurrency or increase timeouts 4) If behind a proxy/LB - Ensure the health check points to
/api/ready
- Confirm upstream points to the Prefect API service/port and that backends are marked healthy Built-in retries - The Prefect client will automatically retry 503s with exponential backoff and jitter. Persistent 503s indicate an underlying server/DB issue that needs fixing. Docs - Health check: https://docs.prefect.io/v3/api-ref/rest-api/server/root/health-check/ - Readiness check: https://docs.prefect.io/v3/api-ref/rest-api/server/root/perform-readiness-check/ - Settings reference: https://docs.prefect.io/v3/api-ref/settings-ref/ - Client retry settings: https://docs.prefect.io/v3/api-ref/python/prefect-settings-models-client/ - Network access: https://docs.prefect.io/v3/advanced/configure-network-access/ Could you share: - The exact
PREFECT_API_URL
value you’re using (sanitized if needed) - Whether you’re running via Docker/Kubernetes/bare metal and which database - The output of
curl -v "$PREFECT_API_URL/ready"
from the worker host - Any recent server logs around startup/migrations If logs are convenient, feel free to upload the last ~200 lines from your Prefect server and (if applicable) database container/pod.