Hi, team Prefect!

Failed to set task state with error: ClientError([{'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.'}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1839, in set_task_run_state
    result = self.graphql(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 563, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.'}}}]

There is a problem with launching tasks in Prefect, on the screenshots, the last two unsuccessful launches returned an error (described above), the last launch was performed manually, ended in success.

Was this flow re-registered during a run? Or did maybe two agents pick up this flow?

No, but I noticed that during registration, information was displayed in the console that the third version was registered, on the UI the fourth latest version. Could this be the reason?

Not sure I follow, but yes I think registration can cause this. Can test more. What is your RunConfig?

I didn’t quite understand the question

Is the Flow running on Kubernetes/Docker/Local?

in Docker

Ok so I tried re-registering and the already running flow just ran to completion. I think the cause of this error is something like the flow is cancelled or was marked as failed in the UI but it’s still running. The flow has a terminal state in the database (not running), but the task is continuing and then tries to update its state, which throws this error saying it can’t update the state because the Flow is not running anymore

The screen shots are not telling me so much. Are all of the failures you are showing related to Internal Server Error?

No, Kevin not everyone is related to this error, the rest have an error related to a duplicate insert in the target storage

Sorry I’m a bit lost now 😅. Are you showing me they are stuck in schedulling?

No :) Look at the latest version in the UI and in the console. I'm sorry that I dispersed you

Ah I see what you mean

Umm, that's something I'll have to find out. Kevin say, Prefect server/agent in RunConfig Docker application uses UTC and GMT + TZ to capture actions?

Prefect backend is all in UTC, and then the UI converts to the timezone you are in. But you can also give Prefect timezone-aware timestamps

The transaction of the first launch in target DWH is now rolling back the transaction

My Russian is a bit rusty 😅 can you describe your problem in English?

sorry

generally speaking, I saw a similar issue with long-running jobs. Do you happen to run some subprocess or Databricks job within your flow that fails with the original error message? Could you share your flow code?

Anna, wir verwenden Perfect Running Docker-Container in Kubernetes

If you want to turn off heartbeats, I suggest adding it on the Flow RunConfig,

flow.run_config = KubernetesRun(...,env={"PREFECT_CLOUD__HEARTBEAT_MODE": "thread"})

it worked, thanks guys!