Hola :slightly_smiling_face: I have an issue with ...
# prefect-community
t
Hola 🙂 I have an issue with a pod not terminating while the flow is finished. ⬇️
There was an exception in a task and then the following logs:
Copy code
INFO - prefect.CloudTaskRunner | Task 'accounts.link': Finished task run for task with final state: 'Retrying'
WARNING - prefect.CloudFlowRunner | Flow run is no longer in a running state; the current state is: <Success: "All reference tasks succeeded.">
[WARNING] Flow run is no longer in a running state; the current state is: <Success: "All reference tasks succeeded.">
WARNING - prefect.CloudFlowRunner | Flow run is no longer in a running state; the current state is: <Success: "All reference tasks succeeded.">
...
ERROR - prefect.CloudTaskRunner | Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID c9cc30f8-ed56-4def-a2d3-7a8c6e882785: provided a running state but associated flow run 78b75834-d31f-4660-ac0a-d7f365f4f2eb is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
INFO - prefect.CloudTaskRunner | Task 'accounts.link': Finished task run for task with final state: 'ClientFailed'
...
Then the flow was manually restarted and finished, however the pod that ran the flow until the exception before the manual retry never died. And strangely in the schematic view, we can see that the task is still in the retrying orange state
If someone has an idea about what is happening and how to fix it ^^ 🙂
k
The fix I think is to delete the pod and mark this as Failed. Still digesting what may have happened here
t
Any idea of what could cause this ? Actually we have every week (our flows runs every weekends/start of the week) several flow pods that are still running while their flow are finished