11/29/2022, 11:19 PM
Hi there, I am seeing this error frequently and I am not sure if it's something I should worry about. Does anybody know how to avoid that error? Prefect server 1.3.1 deployed on GCP Kubernetes cluster
State update failed: this run has already been submitted.

insertId: "c39wn56hnd9wbq96"
labels: {3}
resource: {
labels: {
cluster_name: xxx
container_name: "graphql"
location: "us-central1"
namespace_name: "prefect"
pod_name: "prefect-server-graphql-5d768bcbb5-5nw8c"
project_id: xxx
type: "k8s_container"
severity: "ERROR"
textPayload: "State update failed: this run has already been submitted."
timestamp: "2022-11-15T12:37:21.941099253Z"
It is always followed by another error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/", line 668, in await_completed
    return await completed
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/", line 653, in await_result
    return_type, field_nodes, info, path, await result
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/", line 727, in complete_value
    raise result
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/", line 626, in await_result
    return await result
  File "/prefect-server/src/prefect_server/graphql/", line 52, in resolve
    result = await result
  File "/prefect-server/src/prefect_server/graphql/", line 45, in resolve_set_flow_run_states
    *[check_size_and_set_state(state_input) for state_input in input["states"]]
  File "/prefect-server/src/prefect_server/graphql/", line 39, in check_size_and_set_state
  File "/prefect-server/src/prefect_server/api/", line 63, in set_flow_run_state
    "State update failed: this run has already been submitted."
ValueError: State update failed: this run has already been submitted.

Bianca Hoch

12/02/2022, 3:04 PM
Hello Mo, to my understanding, this could be a case of Kubernetes and Prefect being out of sync. Kubernetes may be attempting to retry the workflow, which has already been submitted by Prefect. Prefect cannot control ALL of the actions Kubernetes takes. If this is the case, there are two recommended actions you could take to remediate this. 1. Disable GKE Autopilot if it is already enabled. 2. Enabling version locking for the flow being impacted
For context, version locking only allows the flow to be run once.


12/04/2022, 11:47 PM
Thanks Bianca. Our cluster is indeed an autopilot one. I don't think it's possible to turn the autopilot feature off. We will need to create another standard cluster I guess. I will keep you posted if this solves the issue.