<@ULVA73B9P> We run a self-hosted Prefect (3.4.6) ...
# ask-marvin
l
@Marvin We run a self-hosted Prefect (3.4.6) with a PostgreSQL backend; we notice that whenever we initiate a quick run via the UI, flows seem to run without any issue, but scheduled runs (either through a defined schedule or custom runs) will just hang in a running state until we cancel it manually. Why?
m
thought for 955.2 seconds
Short answer: this almost always comes down to how the run is executed and whether the runtime can talk back to your self‑hosted API. If the infra starts but the flow process can’t reach the API to send heartbeats/state updates, the run will sit in Running indefinitely. Quick runs can appear to “work” if they’re executing on different infrastructure (or locally) that has API access, while scheduled runs are executed somewhere that doesn’t. Here’s a focused checklist to pinpoint the issue: 1) Confirm what is actually executing the runs - Are both your Quick runs and your scheduled runs coming from the same deployment and work pool? - In the UI, open a flow run → Details → Infrastructure/Worker. Compare a Quick run that completes vs a scheduled run that hangs. Are they using the same work pool and base job template? Docs: - Work pools overview - Schedules and state transitions 2) Verify a worker is polling and picking up the scheduled runs - Make sure you have a worker running for the pool and it’s healthy (polling). - Start a worker with visible logs:
Copy code
prefect worker start -p <your-pool-name> --name debug-worker
- Watch the worker logs as the scheduled run transitions. You should see it pick the run, submit infra, and stream logs. Docs: - Worker healthchecks 3) If a run reaches Running but shows no flow logs, test API connectivity from the runtime This symptom strongly suggests the flow process started but can’t talk to your API to send heartbeats or final states. - For Docker work pools, exec into the container that’s running your flow; for Kubernetes, exec into the job pod. Then:
Copy code
echo $PREFECT_API_URL
curl -sv $PREFECT_API_URL/api/health
- If that curl fails or resolves to the wrong host, fix your network/DNS/firewall or the URL you inject into the job. Ensure the job template sets
PREFECT_API_URL
to a routable address from the worker/pod’s network (not just from your browser or the UI host). Self-hosting notes: - Self-hosted guide 4) Compare base job templates and env for Quick vs scheduled - If Quick runs finish but scheduled runs hang, they may be using different infra. For example: - Quick run executes on a local/process worker (API reachable) - Scheduled run executes on Kubernetes/Docker (API not reachable) - Ensure the same
PREFECT_API_URL
(and any proxies, TLS, or base paths) are applied to the scheduled run’s job template. 5) Look for concurrency gates or paused pools - If you set deployment or work pool concurrency, a run would usually stay in Scheduled (AwaitingConcurrencySlot), not Running — but it’s worth verifying you didn’t accidentally gate the run.
Copy code
prefect work-pool inspect <your-pool-name>
6) Check time/clock skew - Rare, but if clocks are off across servers/containers, state transitions can be odd. Ensure all hosts are NTP‑synced. 7) As a safety net, enable “zombie” detection - This won’t fix connectivity, but it will auto-crash runs that stop heartbeating instead of leaving them stuck in Running: - Detect zombie flows If you can share a bit more, I can be very specific: - What work pool type are you using (process, docker, kubernetes, etc.)? - Do hanging runs show any flow logs at all? - What is the value of
PREFECT_API_URL
inside the running container/pod? - Do worker logs show the run starting and then going silent? - Are Quick runs and scheduled runs triggered from the same deployment? Once we confirm the infra and API reachability are identical between the two, we’ll be able to zero in on the exact fix.