Hi there I am seeing this error frequently and I am not sure

Question

Hi there I am seeing this error frequently and I am not sure if it s something I should worry about Does anybody know how to avoid that error Prefect server 1 3 1 deployed on GCP Kubernetes cluster ```graphql State update failed this run has already been submitted insertId c39wn56hnd9wbq96 labels 3 resource labels cluster name xxx container name graphql location us central1 namespace name prefect pod name prefect server graphql 5d768bcbb5 5nw8c project id xxx type k8s container severity ERROR textPayload State update failed this run has already been submitted timestamp 2022 11 15T12 37 21 941099253Z ```

Mo · Answer

It is always followed by another error ```graphql Traceback most recent call last File usr local lib python3 7 site packages graphql execution execute py line 668 in await completed return await completed File usr local lib python3 7 site packages graphql execution execute py line 653 in await result return type field nodes info path await result File usr local lib python3 7 site packages graphql execution execute py line 727 in complete value raise result File usr local lib python3 7 site packages graphql execution execute py line 626 in await result return await result File prefect server src prefect server graphql extensions py line 52 in resolve result = await result File prefect server src prefect server graphql states py line 45 in resolve set flow run states check size and set state state input for state input in input states File prefect server src prefect server graphql states py line 39 in check size and set state agent id=agent id File prefect server src prefect server api states py line 63 in set flow run state State update failed this run has already been submitted ValueError State update failed this run has already been submitted ```

Bianca Hoch · Answer

Hello Mo to my understanding this could be a case of Kubernetes and Prefect being out of sync Kubernetes may be attempting to retry the workflow which has already been submitted by Prefect Prefect cannot control ALL of the actions Kubernetes takes If this is the case there are two recommended actions you could take to remediate this 1 Disable if it is already enabled 2 Enabling for the flow being impacted

Bianca Hoch · Answer

For context version locking only allows the flow to be run once

Mo · Answer

Thanks Bianca Our cluster is indeed an autopilot one I don t think it s possible to turn the autopilot feature off We will need to create another standard cluster I guess I will keep you posted if this solves the issue