https://prefect.io logo
Join Slack
Powered by
Hi community, how do I fix this `ClientError` mess...
# prefect-community
b
Hi community, how do I fix this 
ClientError
message? All upstream tasks succeeded. Setup: 
prefect 1.2.2
, prefect cloud.
Copy code
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID f0614b56-3c3b-41f6-9481-878c9f4bee78: provided a running state but associated flow run 8647f325-a45d-4124-9995-b68d43faaf15 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 1604, in set_task_run_state
    result = self.graphql(
  File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 464, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID f0614b56-3c3b-41f6-9481-878c9f4bee78: provided a running state but associated flow run 8647f325-a45d-4124-9995-b68d43faaf15 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
✅ 1
b
Hi Beizhen, we'll take a look at this for you. How often does this error occur? Is it for only one flow, or has it affected multiple?
b
Hi Bianca, thanks. This was the first time I encountered the error. As far as I know, it has only affected this one flow. The flow structure is like:
Copy code
with Flow("myflow") as flow:
    param_a = Parameter("param_a", default=["a", "b", "c", "d"])
    param_bool = Parameter("param_bool", default=False)
    mapped_df = get_df.map(mapped_a) # get some data from database
    mapped_df_out = predict_df.map(df, mapped_a, unmapped(param_bool)) # predict in azure ml cluster
    update_table(mapped_df_out)
In the schematic of the failed flow run, 
get_df.map()
all mapped runs completed, the status of the following task 
predict_df()
is `pending`(where error occured), and the last task 
update_table()
is 
triggerfailed
. Let me know if you want more detailed LOGS of the flow. (ps. this flow is set up to run on one worker, i.e. sequentially even tho mapped due to some other issues.)
b
Hi Beizhen, thanks for your patience. This error can happen if a task is submitted to an executor, like dask, and it tries to execute after a flow has already been marked as failed from another task. The other reason is if the infrastructure, like k8s, reschedules a job in the event of pod death and the flow is already marked failed.
After looking at the logs, it looks like this was the sequence of events:
{"id":"41174d30-d0ee-4ba6-88f0-baa9297ef70c","name":"prefect.CloudTaskRunner","timestamp":"2022-08-14T12:44:12.603258+00:00","message":"Task 'predict_valid': Handling state change from Pending to Failed"}
{"id":"ba0161b5-5855-49b9-afeb-d093a6aedec3","name":"prefect.CloudTaskRunner","timestamp":"2022-08-14T12:44:12.780028+00:00","message":"Task 'predict_valid': Finished task run for task with final state: 'Failed'"}
{"id":"14a99c1b-a2f2-4833-8748-09a25bab40d4","name":"prefect.CloudFlowRunner","timestamp":"2022-08-14T12:44:13.328122+00:00","message":"Flow run FAILED: some reference tasks failed."}
And then there were a series of other logs where the tasks attempted to run downstream, accompanied by the errors like the one you mentioned above: 
{"id":"34664978-28a3-4ed8-827a-2448de3d1a01","name":"prefect.CloudTaskRunner","timestamp":"2022-08-14T12:44:16.908438+00:00","message":"Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 7ac8ea1f-94da-475a-b87b-043b82734886: provided a running state but associated flow run 8647f325-a45d-4124-9995-b68d43faaf15 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])\nTraceback (most recent call last):\n File \"/root/miniconda3/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py\", line 91, in call_runner_target_handlers\n  state = self.client.set_task_run_state(\n File \"/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py\", line 1604, in set_task_run_state\n  result = self.graphql(\n File \"/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py\", line 464, in graphql\n  raise ClientError(result[\"errors\"])\nprefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 7ac8ea1f-94da-475a-b87b-043b82734886: provided a running state but associated flow run 8647f325-a45d-4124-9995-b68d43faaf15 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]"}
Generally, these errors aren't going to be a systemic issue.
b
It sounds right. Thanks a lot for clearing it up Bianca!
✨ 1
Hi Bianca The same error happened again since last. Can you help me figuring out how I could avoid it from happening? Thanks 🙂
Copy code
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID c82ddae8-13f9-4fed-b389-7e40e355a5cd: provided a running state but associated flow run 3f4f7066-9acd-41c6-b17e-05219fafb33f is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 1604, in set_task_run_state
    result = self.graphql(
  File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 464, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID c82ddae8-13f9-4fed-b389-7e40e355a5cd: provided a running state but associated flow run 3f4f7066-9acd-41c6-b17e-05219fafb33f is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
5 Views
Open in Slack
PreviousNext
Powered by