Here is a flow_run_id that leads to a flow stuck like that: bd95ac1b-89c3-497c-bf66-272a918e1be0

Another one if it's of any help to investigate: c92e008e-aff5-41c5-ae72-1f10694ad7f7 🙂

but it's seems that flows get stuck on any kind of tasks. In the first one it's on a task with concurrency limit but not for the second one...

?

In the meantime, here is an article that will help alleviate the hangups in the interim while we investigate. These queries can be used to clear out runs that are stuck in a transient state (Running or Cancelling) https://discourse.prefect.io/t/how-can-i-remove-flow-task-runs-stuck-in-a-running-or-canceling-state/1855

I had to restart to the first flow. Do you want me to keep the second one in the "stuck" step while you're investigating ?

Thanks for the additional context! If you want to leave that flow as 'stuck', it would help. Unless it is blocking your production workflows, in that case definitely clear it out using the steps outlined in that article.

we enable this feature to avoid having the same tasks running in duplicate

Hello bianca, to follow on: Here are some flows that were affected by the issue: • flow 6e2217a0-5771-401f-a17d-636751d7acf1 at task: b8b2d8e8-fcc7-4a3b-a998-741264b45f23?logs. This flow is finished, but we can clearly see in the task logs, that the task was running and at some point. Prefect kind of restart the task:

Task 'accounts.link': Starting task run...

and then we have a strange log:

Task 'accounts.link': Finished task run for task with final state: 'Running'

And then we had to manually restart it... • flow: 4b2ed23e-32fd-4794-bdf3-4a05e8b27869 at task: 9c16ce6e-7cea-4edb-96d1-8cfe21bd9813. This one isn't resumed yet, so the stuck task still appears as "running" • Also some time we have another strange log:

INFO CloudTaskRunner Task 'accounts.link_ngram': Finished task run for task with final state: 'ClientFailed'

I can't find the flow run ids now, but we had a few of them. And each time the flow doesn't fail.

Another one 🙂 • flow: 5d3a9f91-1f87-49d0-8bb8-0d3848d656e3 at task: 092a3f27-db56-4dea-aa47-ad11fafb8cd2

and: • flow 8715221f-a8a8-4e92-852b-c8d460623d20 at task c3384b23-4193-4b24-9a4d-228bfdb270bf

up

Looking at the logs of the flow runs and task runs, I'm seeing duplicated records for different events in the logs that are generating seconds apart

Also, to be thorough, I'd suggest updating the additional context section of the GitHub issue as well. I know it's a bit redundant. but it will help with troubleshooting.

Cross posting here just for reference: https://github.com/PrefectHQ/prefect/issues/7712

Just noticed that another user referenced this post, as they were having a similar issue. Mason's explanation of why flows get stuck in a Running state is a good one.

Thanks for the link to the other thread. Actually we use to have issue like the one with "no heartbeat detected", so I know that if for any reason something happens to the pod running the flow, then it's complicated for prefect to know about it. We used to have this issue with pod rescheduled due to autoscaling. But I don't think the current issues is related to this, because I don't see any events of k8s doing anything with the pods, plus there is the weird "restarting" logs. But I'm going to monitor the agent, because the fact that it restarted is a bit strange to me.