Hi everyone! We faced a situation yesterday. A lot of flows were stuck on a subflow for around 2 hrs. They got triggered almost at the same time and we couldn't find any error in the logs as to why this is happening. We use Prefect ECS agent on AWS, and in the agent logs also there was no error. But one thing I noticed was the Prefect UI was showing that the flow was stuck at a subflow, but when I looked into the agent logs the same subflow was already executed and it moved on to the next task. There were no errors there also. Any ideas on why it was stuck at some point but didn't throw any error?
Second is we tried to cancel all the flows that were stuck, but all were stuck in cancelling stage. I checked the agent logs also and it said
Unable to kill process The process is running on a different host
Why this is happening? What could be the cause of this issue?
Any help appriciated
Thanks