Beizhen
08/15/2022, 7:48 AMClientError
message? All upstream tasks succeeded.
Setup: prefect 1.2.2
, prefect cloud.
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID f0614b56-3c3b-41f6-9481-878c9f4bee78: provided a running state but associated flow run 8647f325-a45d-4124-9995-b68d43faaf15 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 1604, in set_task_run_state
result = self.graphql(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 464, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID f0614b56-3c3b-41f6-9481-878c9f4bee78: provided a running state but associated flow run 8647f325-a45d-4124-9995-b68d43faaf15 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
Bianca Hoch
08/15/2022, 2:03 PMBeizhen
08/16/2022, 8:29 AMwith Flow("myflow") as flow:
param_a = Parameter("param_a", default=["a", "b", "c", "d"])
param_bool = Parameter("param_bool", default=False)
mapped_df = get_df.map(mapped_a) # get some data from database
mapped_df_out = predict_df.map(df, mapped_a, unmapped(param_bool)) # predict in azure ml cluster
update_table(mapped_df_out)
In the schematic of the failed flow run, get_df.map()
all mapped runs completed, the status of the following task predict_df()
is `pending`(where error occured), and the last task update_table()
is triggerfailed
.
Let me know if you want more detailed LOGS of the flow.
(ps. this flow is set up to run on one worker, i.e. sequentially even tho mapped due to some other issues.)Bianca Hoch
08/17/2022, 5:02 PM{"id":"41174d30-d0ee-4ba6-88f0-baa9297ef70c","name":"prefect.CloudTaskRunner","timestamp":"2022-08-14T12:44:12.603258+00:00","message":"Task 'predict_valid': Handling state change from Pending to Failed"}
{"id":"ba0161b5-5855-49b9-afeb-d093a6aedec3","name":"prefect.CloudTaskRunner","timestamp":"2022-08-14T12:44:12.780028+00:00","message":"Task 'predict_valid': Finished task run for task with final state: 'Failed'"}
{"id":"14a99c1b-a2f2-4833-8748-09a25bab40d4","name":"prefect.CloudFlowRunner","timestamp":"2022-08-14T12:44:13.328122+00:00","message":"Flow run FAILED: some reference tasks failed."}
{"id":"34664978-28a3-4ed8-827a-2448de3d1a01","name":"prefect.CloudTaskRunner","timestamp":"2022-08-14T12:44:16.908438+00:00","message":"Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 7ac8ea1f-94da-475a-b87b-043b82734886: provided a running state but associated flow run 8647f325-a45d-4124-9995-b68d43faaf15 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])\nTraceback (most recent call last):\n File \"/root/miniconda3/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py\", line 91, in call_runner_target_handlers\n state = self.client.set_task_run_state(\n File \"/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py\", line 1604, in set_task_run_state\n result = self.graphql(\n File \"/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py\", line 464, in graphql\n raise ClientError(result[\"errors\"])\nprefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 7ac8ea1f-94da-475a-b87b-043b82734886: provided a running state but associated flow run 8647f325-a45d-4124-9995-b68d43faaf15 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]"}
Beizhen
08/23/2022, 6:55 AMFailed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID c82ddae8-13f9-4fed-b389-7e40e355a5cd: provided a running state but associated flow run 3f4f7066-9acd-41c6-b17e-05219fafb33f is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 1604, in set_task_run_state
result = self.graphql(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 464, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID c82ddae8-13f9-4fed-b389-7e40e355a5cd: provided a running state but associated flow run 3f4f7066-9acd-41c6-b17e-05219fafb33f is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]