Hi guys I am trying to set my prefect flow to use temporary

Question

Hi guys I am trying to set my prefect flow to use temporary dask clusters in GCP but it fails most of the time and the latest error is ```prefect exceptions ClientError path set task run states message State update failed for task run ID e18b41ed 7ce8 449f bda2 6de3956eb555 provided a running state but associated flow run 8484f5dc acef 4ac5 ae08 9bf6d72edcf2 is not in a running state extensions code INTERNAL SERVER ERROR ``` cloud you guys help me understand what causes this issue

Bianca Hoch · Answer

Hey Miremad I think what happened here is that the flow entered a `Failed` state and the process that was running this task attempted to keep running it after the fact Looking at the logs and timestamps I see the following The flow fails with this message ```2023 03 17T04 38 14 975414+00 00 Unexpected error lt HttpError 404 when requesting <https compute googleapis com compute v1 projects chemix ai qa zones us west1 a instances dask 68d8ddad worker e3ad8d71 alt=json> returned The resource projects chemix ai qa zones us west1 a instances dask 68d8ddad worker e3ad8d71 was not found Details message The resource projects chemix ai qa zones us west1 a instances dask 68d8ddad worker e3ad8d71 was not found domain global reason notFound gt ```

Bianca Hoch · Answer

Then the task from the message you shared throws this error ```2023 03 17T04 38 16 668754+00 00 Task train and predict on fold 3 Finished task run for task with final state ClientFailed ```

Bianca Hoch · Answer

Followed by ```2023 03 17T04 38 16 648918+00 00 Failed to set task state with error ClientError path set task run states message State update failed for task run ID e18b41ed 7ce8 449f bda2 6de3956eb555 provided a running state but associated flow run 8484f5dc acef 4ac5 ae08 9bf6d72edcf2 is not in a running state extensions code INTERNAL SERVER ERROR Traceback most recent call last File usr local lib python3 10 site packages prefect engine cloud task runner py line 91 in call runner target handlers state = self client set task run state File usr local lib python3 10 site packages prefect client client py line 1622 in set task run state result = self graphql File usr local lib python3 10 site packages prefect client client py line 465 in graphql raise ClientError result errors prefect exceptions ClientError path set task run states message State update failed for task run ID e18b41ed 7ce8 449f bda2 6de3956eb555 provided a running state but associated flow run 8484f5dc acef 4ac5 ae08 9bf6d72edcf2 is not in a running state extensions code INTERNAL SERVER ERROR ```

Miremad Aghili · Answer

Hi Bianca thanks for your help but why would that enter a failed state for worker not found shouldn t it be following what workers are around and what workers are dead