Anish S
05/08/2026, 1:37 PM/api/health always responds true even when database password has changed and other apis are returning 500 Internal Server Error. Why? What endpoint should I use for healthcheck?Marvin
05/08/2026, 1:37 PMMarvin
05/08/2026, 1:38 PM/api/health is just a liveness check in Prefect 3.x — it always returns true if the API process is up.
In the server code, it is literally implemented as:
@api_app.get(health_check_path, tags=["Root"])
async def health_check() -> bool:
return True
So it does not verify:
- database connectivity
- migrations
- API dependency health
That’s why you can see:
- /api/health -> 200 true
- other API routes -> 500 Internal Server Error
if the database password changed or the DB is otherwise unavailable.
Use /api/ready for readiness/real healthchecks.
/api/ready checks database connectivity via db.is_db_connectable() and returns:
- 200 {"message":"OK"} when DB is reachable
- 503 {"message":"Database is not available"} when it is not
So the practical guidance is:
- Liveness probe: /api/health
- Readiness probe / load balancer healthcheck: /api/ready
Example Kubernetes probes:
livenessProbe:
httpGet:
path: /api/health
port: 4200
readinessProbe:
httpGet:
path: /api/ready
port: 4200
Relevant source references:
- `/api/health` implementation
- `/api/ready` implementation
- database connectivity check
One note: there appears to be some docs mismatch around /api/health response shape, but the source code is clear: it returns boolean true.
If you want, I can also help you map this into a Docker Compose, Kubernetes, or ALB/NLB healthcheck configuration.Anish S
05/08/2026, 2:22 PM/api/ready still. The prefect server has a mixed responses. partly responding with 200 and 500 for several api requests (example: api/flow_runs/count). Why is that? Is the server caching or something?Marvin
05/08/2026, 2:22 PMMarvin
05/08/2026, 2:25 PM/api/ready checks is much narrower than what a real API request does.
Why /api/ready can be 200 while /api/flow_runs/count is 500
/api/ready only tests whether Prefect can obtain a database connection:
async def is_db_connectable(self) -> bool:
engine = await self.engine()
try:
async with engine.connect():
return True
except Exception:
return False
That means it answers roughly:
“Can I open a DB connection right now?”It does not prove: - a session can be created successfully - a transaction can run successfully - a real query against Prefect tables will succeed - all replicas/pods behind a load balancer are healthy - the database is free of locks / deadlocks / partial outages By contrast,
/api/flow_runs/count does a real DB session + query against Prefect tables, so it can fail even when /api/ready succeeds.
Relevant source:
- `/api/ready`
- `is_db_connectable()`
- `/api/flow_runs/count` implementation
Common reasons for “mixed” 200 and 500 responses
1. Connection succeeds, query fails
- /api/ready only opens a connection.
- /api/flow_runs/count runs a real SQL query.
- So issues like locks, bad transaction state, schema mismatch, or query-time DB errors can show up only on the real endpoint.
2. Intermittent/stale pooled connections
- Prefect uses SQLAlchemy connection pooling.
- A simple connect/ping can succeed, while a later query on a pooled connection can still fail in practice.
3. Multiple API replicas behind a load balancer
- One pod may be healthy enough to answer /api/ready.
- Another pod may be broken and returning 500 for some requests.
- This is a very common cause of “sometimes 200, sometimes 500”.
4. Database partial availability
- The DB may accept connections but have problems executing queries on specific tables, under lock contention, or during failover/recovery.
5. Startup/migration timing
- The process may be up and connectable before it is fully able to serve all API requests.
So: is there a better endpoint?
Prefect does not currently provide a built-in endpoint that fully validates “all API operations are healthy”.
/api/ready is still the best built-in readiness check, but it is only a lightweight DB-connectivity check.
If you need a stronger health check for your environment, use a synthetic request that exercises a real API operation, for example something like:
- POST /api/flow_runs/count
- or another low-cost endpoint that touches the DB in the same way your app depends on
That gives you an application-level healthcheck rather than just a connectivity healthcheck.
What I’d check next
If you’re seeing mixed 200/500 responses, I’d suspect one of these first:
- multiple Prefect API instances/pods with inconsistent health
- DB failover / replica / proxy issues
- connection pool issues after DB credential or endpoint changes
- migration/schema mismatch
- database locks or resource exhaustion
A useful diagnostic is:
- hit /api/ready and /api/flow_runs/count repeatedly from the same client
- compare timestamps with Prefect API logs
- if running multiple replicas, check whether failures correlate to a specific pod/container
If you want, I can help you narrow this down for your setup — e.g. Docker Compose, Kubernetes, ECS, Helm, RDS/CloudSQL/Postgres, number of API replicas, and how the DB password was changed.