Howdy folks - I’ve been getting all flow runs scheduled in the last 15 minutes or so showing up as Late or Scheduled always. I’m using an ECS push work pool, and the pool itself appears to be healthy. Furthermore, the ECS tasks appear to start and run OK, but for whatever reason Prefect doesn’t seem to think that these jobs have been scheduled. How can I best debug this issue?

Hey Krishna, sorry about the friction there. The best way forward is to try to create a minimal reproducible example and file a github issue here.

Hey, thanks for the response Will. In this case I’m not reporting a code bug but an ongoing operational issue - there’s not really a reproducible example I can provide for that. But many of my flows are left stuck in either the Late or Pending states (like the sea of gray in the middle of the screenshot here).

I'll take a look and see if any others have experiences it or if we see any anomalies on our side. Thanks.

Hey Krishnan - do you have a concurrency limit set on your work pool? Is it possible those pending runs are taking up your slots?

Hey Jenny! No concurrency limit on the pool, which is what surprises me

👀

Still seeing the issue today, as you can see with the cluster of gray on the left. Just to confirm the concurrency limit, I ran:

prefect work-pool ls
                                         Work Pools
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃ Name               ┃ Type     ┃                                   ID ┃ Concurrency Limit ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ ingest-pool-public │ ecs:push │ REDACTED                             │ None              │
│ ingest-pool        │ ecs:push │ REDACTED                             │ None              │
└────────────────────┴──────────┴──────────────────────────────────────┴───────────────────┘

Ah ok! Sorry - it's never the easy fix 😆 As Will said we'll take a look and see if any others have experiences it or if we see any anomalies on our side. Thanks.

Hey @Krishnan Chandra - i cant find any anomalies or other reports of this, would you give me your acccount id and any affected flow run IDs and I can ask our on-call engineer to see if we got any logs back from ECS? Thanks.