Mohamed Zaki
08/21/2025, 2:55 AMlate
flow runs. I sent a lot of flow runs to a worker pool, some of old flows ~8 hours ago stayed in late status, and pool executes new flows ~1 min ago and leaves old flows. Isn't it supposed to be executing in FIFI order?Marvin
08/21/2025, 2:56 AMMarvin
08/21/2025, 3:03 AMprefect flow-run ls --state LATE --limit 50
- Inspect a specific late run (check its work pool/queue, tags, concurrency keys, and any limit messages in the state details):
prefect flow-run inspect <FLOW_RUN_ID>
- Check your work pool config (including pool-level concurrency limit):
prefect work-pool inspect <POOL_NAME>
- List global concurrency limits:
prefect gcl ls
- Confirm your worker is polling the intended pool/queue(s) and check its logs:
prefect worker start -p <POOL_NAME> -q <QUEUE_NAME>
If you need stronger ordering
- Prefect doesn’t guarantee FIFO in a pool. If strict ordering matters, consider:
- Using a single queue per ordered stream of work and avoiding overlapping concurrency limits.
- Serializing with a concurrency key so only one run for a given key executes at a time. This enforces serialization, though it still does not hard-guarantee FIFO across unrelated runs.
- Implementing ordering at the producer layer (e.g., don’t release run N+1 until run N is complete).
Helpful docs
- Workers: Workers
- Work pools: Work pools
- Worker health/polling: Worker health checks
- Global concurrency limits: Global concurrency limits
A couple clarifying questions to pinpoint your case:
- Are the late runs assigned to the exact same work pool and queue(s) your worker is polling?
- Do those late runs share tags or concurrency keys that could be limited by a global or pool-level concurrency limit?
- Do you see any “waiting for concurrency slot” notes in the late runs’ state details or in worker logs?
If you share one late flow run ID and the pool/queue you expect it to use, I can help read the state details and narrow it down.