Hi community how do I fix this `ClientError` message All ups

Question

Hi community how do I fix this `ClientError` message All upstream tasks succeeded Setup `prefect 1 2 2` prefect cloud ```Failed to set task state with error ClientError path set task run states message State update failed for task run ID f0614b56 3c3b 41f6 9481 878c9f4bee78 provided a running state but associated flow run 8647f325 a45d 4124 9995 b68d43faaf15 is not in a running state extensions code INTERNAL SERVER ERROR Traceback most recent call last File root miniconda3 lib python3 8 site packages prefect engine cloud task runner py line 91 in call runner target handlers state = self client set task run state File root miniconda3 lib python3 8 site packages prefect client client py line 1604 in set task run state result = self graphql File root miniconda3 lib python3 8 site packages prefect client client py line 464 in graphql raise ClientError result errors prefect exceptions ClientError path set task run states message State update failed for task run ID f0614b56 3c3b 41f6 9481 878c9f4bee78 provided a running state but associated flow run 8647f325 a45d 4124 9995 b68d43faaf15 is not in a running state extensions code INTERNAL SERVER ERROR ```

Bianca Hoch · Answer

Hi Beizhen we ll take a look at this for you How often does this error occur Is it for only one flow or has it affected multiple

Beizhen · Answer

Hi Bianca thanks This was the first time I encountered the error As far as I know it has only affected this one flow The flow structure is like ```with Flow myflow as flow param a = Parameter param a default= a b c d param bool = Parameter param bool default=False mapped df = get df map mapped a get some data from database mapped df out = predict df map df mapped a unmapped param bool predict in azure ml cluster update table mapped df out ``` In the schematic of the failed flow run `get df map ` all mapped runs completed the status of the following task `predict df ` is `pending` where error occured and the last task `update table ` is `triggerfailed` Let me know if you want more detailed LOGS of the flow ps this flow is set up to run on one worker i e sequentially even tho mapped due to some other issues

Bianca Hoch · Answer

Hi Beizhen thanks for your patience This error can happen if a task is submitted to an executor like dask and it tries to execute after a flow has already been marked as failed from another task The other reason is if the infrastructure like k8s reschedules a job in the event of pod death and the flow is already marked failed

Bianca Hoch · Answer

After looking at the logs it looks like this was the sequence of events

Bianca Hoch · Answer

` id 41174d30 d0ee 4ba6 88f0 baa9297ef70c name prefect CloudTaskRunner timestamp 2022 08 14T12 44 12 603258+00 00 message Task predict valid Handling state change from Pending to Failed ` ` id ba0161b5 5855 49b9 afeb d093a6aedec3 name prefect CloudTaskRunner timestamp 2022 08 14T12 44 12 780028+00 00 message Task predict valid Finished task run for task with final state Failed ` ` id 14a99c1b a2f2 4833 8748 09a25bab40d4 name prefect CloudFlowRunner timestamp 2022 08 14T12 44 13 328122+00 00 message Flow run FAILED some reference tasks failed `

Bianca Hoch · Answer

And then there were a series of other logs where the tasks attempted to run downstream accompanied by the errors like the one you mentioned above ` id 34664978 28a3 4ed8 827a 2448de3d1a01 name prefect CloudTaskRunner timestamp 2022 08 14T12 44 16 908438+00 00 message Failed to set task state with error ClientError path set task run states message State update failed for task run ID 7ac8ea1f 94da 475a b87b 043b82734886 provided a running state but associated flow run 8647f325 a45d 4124 9995 b68d43faaf15 is not in a running state extensions code INTERNAL SERVER ERROR nTraceback most recent call last n File root miniconda3 lib python3 8 site packages prefect engine cloud task runner py line 91 in call runner target handlers n state = self client set task run state n File root miniconda3 lib python3 8 site packages prefect client client py line 1604 in set task run state n result = self graphql n File root miniconda3 lib python3 8 site packages prefect client client py line 464 in graphql n raise ClientError result errors nprefect exceptions ClientError path set task run states message State update failed for task run ID 7ac8ea1f 94da 475a b87b 043b82734886 provided a running state but associated flow run 8647f325 a45d 4124 9995 b68d43faaf15 is not in a running state extensions code INTERNAL SERVER ERROR `

Bianca Hoch · Answer

Generally these errors aren t going to be a systemic issue

Beizhen · Answer

It sounds right Thanks a lot for clearing it up Bianca

Beizhen · Answer

Hi Bianca The same error happened again since last Can you help me figuring out how I could avoid it from happening Thanks slightly smiling face ```Failed to set task state with error ClientError path set task run states message State update failed for task run ID c82ddae8 13f9 4fed b389 7e40e355a5cd provided a running state but associated flow run 3f4f7066 9acd 41c6 b17e 05219fafb33f is not in a running state extensions code INTERNAL SERVER ERROR Traceback most recent call last File root miniconda3 lib python3 8 site packages prefect engine cloud task runner py line 91 in call runner target handlers state = self client set task run state File root miniconda3 lib python3 8 site packages prefect client client py line 1604 in set task run state result = self graphql File root miniconda3 lib python3 8 site packages prefect client client py line 464 in graphql raise ClientError result errors prefect exceptions ClientError path set task run states message State update failed for task run ID c82ddae8 13f9 4fed b389 7e40e355a5cd provided a running state but associated flow run 3f4f7066 9acd 41c6 b17e 05219fafb33f is not in a running state extensions code INTERNAL SERVER ERROR ```