<@ULVA73B9P> How does prefect handles `late` flow ...
# marvin-ai
m
@Marvin How does prefect handles
late
flow runs. I sent a lot of flow runs to a worker pool, some of old flows ~8 hours ago stayed in late status, and pool executes new flows ~1 min ago and leaves old flows. Isn't it supposed to be executing in FIFI order?
m
thought for 451.1 seconds
Short answer: Prefect does not guarantee FIFO ordering. “Late” means the run’s scheduled start time has passed, but workers do not inherently prioritize late runs over newer ones. It’s normal to see newer runs start before older late runs depending on pool/queue filters, concurrency limits, and available workers. What “Late” means - A run becomes Late when it wasn’t picked up by its scheduled time. It’s still eligible to run; the state type is effectively “scheduled”, just overdue. - Workers poll the API (about every 15s by default) and claim eligible runs, but there’s no FIFO guarantee across a pool. Common reasons older runs stay Late while newer runs start - Different queues: Your worker may be polling queue A, while older runs are assigned to queue B in the same pool. - Concurrency limits: - Work pool concurrency limit is set and consumed by other runs. - Global concurrency limits (by keys/tags) throttle those older runs while newer ones don’t hit the same limit. - Tag-based filtering or automations that affect eligibility. - Multiple workers concurrently claiming batches can lead to out-of-order starts. How to diagnose quickly - List late runs:
Copy code
prefect flow-run ls --state LATE --limit 50
- Inspect a specific late run (check its work pool/queue, tags, concurrency keys, and any limit messages in the state details):
Copy code
prefect flow-run inspect <FLOW_RUN_ID>
- Check your work pool config (including pool-level concurrency limit):
Copy code
prefect work-pool inspect <POOL_NAME>
- List global concurrency limits:
Copy code
prefect gcl ls
- Confirm your worker is polling the intended pool/queue(s) and check its logs:
Copy code
prefect worker start -p <POOL_NAME> -q <QUEUE_NAME>
If you need stronger ordering - Prefect doesn’t guarantee FIFO in a pool. If strict ordering matters, consider: - Using a single queue per ordered stream of work and avoiding overlapping concurrency limits. - Serializing with a concurrency key so only one run for a given key executes at a time. This enforces serialization, though it still does not hard-guarantee FIFO across unrelated runs. - Implementing ordering at the producer layer (e.g., don’t release run N+1 until run N is complete). Helpful docs - Workers: Workers - Work pools: Work pools - Worker health/polling: Worker health checks - Global concurrency limits: Global concurrency limits A couple clarifying questions to pinpoint your case: - Are the late runs assigned to the exact same work pool and queue(s) your worker is polling? - Do those late runs share tags or concurrency keys that could be limited by a global or pool-level concurrency limit? - Do you see any “waiting for concurrency slot” notes in the late runs’ state details or in worker logs? If you share one late flow run ID and the pool/queue you expect it to use, I can help read the state details and narrow it down.