Hey everyone, I'm running into an issue when roughly 1-2% of flow runs are forever stuck in Pending. They run in Prefect Cloud on an ECS Push work pool. I usually see

RunTask

commands in Cloudtrail whenever Prefect successfully starts new ECS task. But there's nothing logged in Cloudtrail for Pending flow runs, not even an error. Is there a way to debug this issue or turn on more verbose logging?

Hey! Could you share your workspace id and a flow run id that you see stuck in PENDING? feel free to dm if you prefer

👀

@Jake Kaplan sure workspace id =

6985e3e2-dae2-422a-9154-169a004ccfb9

and flow run id =

2c273333-2238-4bd4-b60f-916bfa924fab

Let me know if you need more IDs. I see this issue at least once a day

thanks, if you have a couple more ids just to see if I can identify a pattern that would be helpful. Otherwise will have to do some digging and get back to you

sorry for the delay so, these are still pending:

33753871-addb-4678-887b-75e136d54c9f
7154ccf2-d704-4190-94ab-e70abb8749f5
0eb42a64-f50b-4647-be50-ccb00713153c
2c273333-2238-4bd4-b60f-916bfa924fab
1fce3067-debe-4288-8d77-be7b1519dd82
07ff9e43-89fb-44a1-92d7-4669c90d2708
c74c2ccf-cd91-48a6-a932-9fcc6608353c
083cec02-09eb-45f2-a4df-71698c3f88d9
47d6379a-c27c-476c-8d76-6c4ffa7d9e31
d11af57f-d2b3-4c89-a156-4c346125745f
67f00d4f-acd8-47f5-9eae-118d07e9f269
4927cb76-4395-436b-935f-5efc0a2c6824
855fa155-fd48-4b54-a785-1c545071fee2
692f2e9f-7000-40ab-b487-56151e877d07
db47a80d-7857-425a-a880-72623663a188

and then these - I cancelled them manually few days back

a4747fc6-11c9-4e77-b34b-2fe09d2b5ffb
a0d52646-d7d0-4e82-9fe5-1714b4db89cb
788e9a58-90c1-417e-8be1-52d43f0c3613
c0dd2dae-fe4f-4ef6-9b44-c5c15b82cc95
aa274c78-7cd0-488e-beb2-65050182612d
a1f746e5-0e26-4dd7-b2a4-4e986e411a78
ce10b73a-0f17-46dc-9177-646f1082d694
5412d8eb-4cda-45b8-8752-9d08d1156209
f22682d5-5741-44ce-bdb6-3b123cfe95b1
a018e110-53f0-41d7-8a29-c833331b8fc3
6efc1b62-6f25-42cc-9289-e3665b039495
7af6fd57-f0c6-4585-b1fa-7bd46546bc3f
610bc16c-6c1d-41b3-a02a-10c59eaae887
f6f37e6a-6663-49b7-9d1c-51c80c88e168
6efdd98b-0f09-4164-9447-f71155c40f57
d7cf9eaf-14d5-44a2-b721-5da6c6a47606
69adb7e0-0ed6-41ab-beca-7c9357108f81
95fdc6e0-3912-4665-84e9-594cbab0b8ff
d451a6f2-9f13-4336-89de-4ee9a21938ec
3b98b27f-a22b-45a5-9526-ab38115dc32f
26a0de6f-0faf-4f8b-966e-1f8385932b1d
4dc750ed-0755-4d76-a746-7bd8011acb12
ec9d0ef3-82dd-4317-89f3-6bee11c874cc
e8db5a3f-054e-423c-8b2c-56f86b3f3480
2200dede-39a3-49a7-821b-fe0e044a6577

thanks for looking into this

thanks for sending those! I believe I have an idea of what may be happening here. One thing that would be helpful to verify is that you're not seeing any ECS tasks created for these flow runs correct?

Correct. No ECS tasks. No activity in Cloudtrail

thanks! I am having some difficulty reproducing the exact thing you're experiencing, but I have a hunch. I'm going to enable something and keep an eye on your flow runs for the next 24hr or so to see if this goes away.