<@ULVA73B9P> `/api/health` always responds true ev...
# ask-marvin
a
@Marvin
/api/health
always responds true even when database password has changed and other apis are returning
500 Internal Server Error
. Why? What endpoint should I use for healthcheck?
m
thought for 90.6 seconds
/api/health
is just a liveness check in Prefect 3.x — it always returns
true
if the API process is up. In the server code, it is literally implemented as:
Copy code
@api_app.get(health_check_path, tags=["Root"])
async def health_check() -> bool:
    return True
So it does not verify: - database connectivity - migrations - API dependency health That’s why you can see: -
/api/health
->
200 true
- other API routes ->
500 Internal Server Error
if the database password changed or the DB is otherwise unavailable. Use
/api/ready
for readiness/real healthchecks.
/api/ready
checks database connectivity via
db.is_db_connectable()
and returns: -
200 {"message":"OK"}
when DB is reachable -
503 {"message":"Database is not available"}
when it is not So the practical guidance is: - Liveness probe:
/api/health
- Readiness probe / load balancer healthcheck:
/api/ready
Example Kubernetes probes:
Copy code
livenessProbe:
  httpGet:
    path: /api/health
    port: 4200

readinessProbe:
  httpGet:
    path: /api/ready
    port: 4200
Relevant source references: - `/api/health` implementation - `/api/ready` implementation - database connectivity check One note: there appears to be some docs mismatch around
/api/health
response shape, but the source code is clear: it returns boolean
true
. If you want, I can also help you map this into a Docker Compose, Kubernetes, or ALB/NLB healthcheck configuration.
a
@Marvin I am still getting 200 for
/api/ready
still. The prefect server has a mixed responses. partly responding with
200
and
500
for several api requests (example:
api/flow_runs/count
). Why is that? Is the server caching or something?
m
thought for 150.2 seconds
Yes — this can happen, and it’s not because Prefect is caching the response. What
/api/ready
checks is much narrower than what a real API request does. Why
/api/ready
can be 200 while
/api/flow_runs/count
is 500
/api/ready
only tests whether Prefect can obtain a database connection:
Copy code
async def is_db_connectable(self) -> bool:
    engine = await self.engine()
    try:
        async with engine.connect():
            return True
    except Exception:
        return False
That means it answers roughly:
“Can I open a DB connection right now?”
It does not prove: - a session can be created successfully - a transaction can run successfully - a real query against Prefect tables will succeed - all replicas/pods behind a load balancer are healthy - the database is free of locks / deadlocks / partial outages By contrast,
/api/flow_runs/count
does a real DB session + query against Prefect tables, so it can fail even when
/api/ready
succeeds. Relevant source: - `/api/ready` - `is_db_connectable()` - `/api/flow_runs/count` implementation Common reasons for “mixed” 200 and 500 responses 1. Connection succeeds, query fails -
/api/ready
only opens a connection. -
/api/flow_runs/count
runs a real SQL query. - So issues like locks, bad transaction state, schema mismatch, or query-time DB errors can show up only on the real endpoint. 2. Intermittent/stale pooled connections - Prefect uses SQLAlchemy connection pooling. - A simple connect/ping can succeed, while a later query on a pooled connection can still fail in practice. 3. Multiple API replicas behind a load balancer - One pod may be healthy enough to answer
/api/ready
. - Another pod may be broken and returning 500 for some requests. - This is a very common cause of “sometimes 200, sometimes 500”. 4. Database partial availability - The DB may accept connections but have problems executing queries on specific tables, under lock contention, or during failover/recovery. 5. Startup/migration timing - The process may be up and connectable before it is fully able to serve all API requests. So: is there a better endpoint? Prefect does not currently provide a built-in endpoint that fully validates “all API operations are healthy”.
/api/ready
is still the best built-in readiness check, but it is only a lightweight DB-connectivity check. If you need a stronger health check for your environment, use a synthetic request that exercises a real API operation, for example something like: -
POST /api/flow_runs/count
- or another low-cost endpoint that touches the DB in the same way your app depends on That gives you an application-level healthcheck rather than just a connectivity healthcheck. What I’d check next If you’re seeing mixed 200/500 responses, I’d suspect one of these first: - multiple Prefect API instances/pods with inconsistent health - DB failover / replica / proxy issues - connection pool issues after DB credential or endpoint changes - migration/schema mismatch - database locks or resource exhaustion A useful diagnostic is: - hit
/api/ready
and
/api/flow_runs/count
repeatedly from the same client - compare timestamps with Prefect API logs - if running multiple replicas, check whether failures correlate to a specific pod/container If you want, I can help you narrow this down for your setup — e.g. Docker Compose, Kubernetes, ECS, Helm, RDS/CloudSQL/Postgres, number of API replicas, and how the DB password was changed.