Hello, I updated Server version to 2.10.10 to address the issue that I reported here. Unfortunately some other issue started to come through. My flows are now run 2 time in a row one by one. 1. Prefect 2.10.10 on server & agent(not worker) 2. Run on k8s-job 3. There is only one agent here 4. Flow gets completed and then goes to running again after 5 seconds. a. We limit concurrency on this queue to 40 and we usually get more than 40 flows running so they get late(might this be a hint?)

Do you have on_failure or on_completion hooks? The message shows that there was an error in completing and transitioning to a a

COMPELETED

state, although that might be after your flow code finishes if there are no failures in your tasks. If you’re using kubernetes, I would like at the actual traceback on the pods

I don’t have any hook. The reason is because after successful run we delete data from Redis. So the second run fails. Here are transitions of this flow

2023-05-22T20:12:44.545268+00:00 SCHEDULED Scheduled
2023-05-22T20:12:49.824677+00:00 PENDING Pending
2023-05-22T20:13:03.678826+00:00 RUNNING Running
2023-05-22T20:13:56.621322+00:00 COMPLETED Completed
2023-05-22T20:14:05.842324+00:00 RUNNING Running
2023-05-22T20:29:57.734841+00:00 FAILED Failed

I can see that between PENDING & RUNNING is 14min & pod watch time is 10min. Maybe this can cause scheduling another k8s job?