hello Prefect Support, I have a flow run that, right after it started the first task

init_run

,

Task 'init_run': Starting task run...

it got this error message ( see thread) and, the log message said

Task 'init_run': Finished task run for task with final state: 'ClientFailed'

After that, All remaining tasks stuck in

Pending

state for hours. Any ideas what happened? Maybe my flow need to know how to handle this

ClientFailed

state ?

Hi @Hui Zheng - thanks for the question. It looks like your flow run is/was not in a running state when your task run started. Are you able to check the flow run's logs and see if there's any further info there? Also - would you mind moving the traceback from the main message into this thread? It makes it easier for us to scan through the support channel and make sure we don't miss any messages. Thanks!

for sure, below is the traceback

Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID b5e8b3d8-591e-4165-8bb2-c926157b0b31: provided a running state but associated flow run b3b3514c-9856-447e-b151-238ed9f6c243 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 124, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1453, in set_task_run_state
    result = self.graphql(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 315, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID b5e8b3d8-591e-4165-8bb2-c926157b0b31: provided a running state but associated flow run b3b3514c-9856-447e-b151-238ed9f6c243 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
a