Hello, Frank from Replica here. Yesterday we were having an issue with Flow Runs failing with `API_E...

Frank Hardisty

02/01/2023, 12:24 AM

Hello, Frank from Replica here. Yesterday we were having an issue with Flow Runs failing with `API_ERROR`s. Yesterday we were having similar issues, my colleague @Brett Naul reached out and the issue seemed resolved, however it's happening again. Can anyone advise? Details in thread. Thanks in advance.

✅ 1

Frank Hardisty

02/01/2023, 12:25 AM

Example Flow Run: https://cloud.prefect.io/replica/flow-run/0ab41ce2-38e2-49f7-bd70-ee091aa1f786

Frank Hardisty

02/01/2023, 12:26 AM

Task Run: https://cloud.prefect.io/replica/task-run/b65e2936-eca0-4993-80a5-7a4dfb1a63fd

Frank Hardisty

02/01/2023, 12:27 AM

Message logged:

Copy code

Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/client.py", line 1618, in set_task_run_state
    result = self.graphql(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/client.py", line 464, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]

Frank Hardisty

02/01/2023, 1:12 AM

Update: Brett opened a support case via email, we are getting help through that channel. I'll post the resolution here when we have one.

Frank Hardisty

02/01/2023, 2:05 AM

It turned out to be Task Concurrency. Deleting the references to Task Concurrency in our code was not enough, finally fixing the issue for good required deleting the limits using the Prefect API.

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.