Hola :slightly_smiling_face:
I have an issue with ...
# ask-communityt
Thomas Opsomer
05/27/2022, 1:06 PMHola 🙂
I have an issue with a pod not terminating while the flow is finished. ⬇️
Thomas Opsomer
05/27/2022, 1:10 PMThere was an exception in a task and then the following logs:
Copy code
INFO - prefect.CloudTaskRunner | Task 'accounts.link': Finished task run for task with final state: 'Retrying'
WARNING - prefect.CloudFlowRunner | Flow run is no longer in a running state; the current state is: <Success: "All reference tasks succeeded.">
[WARNING] Flow run is no longer in a running state; the current state is: <Success: "All reference tasks succeeded.">
WARNING - prefect.CloudFlowRunner | Flow run is no longer in a running state; the current state is: <Success: "All reference tasks succeeded.">
...
ERROR - prefect.CloudTaskRunner | Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID c9cc30f8-ed56-4def-a2d3-7a8c6e882785: provided a running state but associated flow run 78b75834-d31f-4660-ac0a-d7f365f4f2eb is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
INFO - prefect.CloudTaskRunner | Task 'accounts.link': Finished task run for task with final state: 'ClientFailed'
...Thomas Opsomer
05/27/2022, 1:13 PMThen the flow was manually restarted and finished, however the pod that ran the flow until the exception before the manual retry never died. And strangely in the schematic view, we can see that the task is still in the retrying orange state
Thomas Opsomer
05/27/2022, 1:14 PMIf someone has an idea about what is happening and how to fix it ^^ 🙂
k
Kevin Kho
05/27/2022, 3:11 PM
The fix I think is to delete the pod and mark this as Failed. Still digesting what may have happened here
t
Thomas Opsomer
06/09/2022, 3:40 PMAny idea of what could cause this ? Actually we have every week (our flows runs every weekends/start of the week) several flow pods that are still running while their flow are finished
3 Views