Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hello, Frank from Replica here. Yesterday we were having an issue with Flow Runs failing with `API_ERROR`s. Yesterday we were having similar issues, my colleague <@UL455076D> reached out and the issue seemed resolved, however it's happening again. Can anyone advise? Details in thread. Thanks in advance.

Example Flow Run: <https://cloud.prefect.io/replica/flow-run/0ab41ce2-38e2-49f7-bd70-ee091aa1f786>

Task Run: <https://cloud.prefect.io/replica/task-run/b65e2936-eca0-4993-80a5-7a4dfb1a63fd>

Message logged:
```Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/client.py", line 1618, in set_task_run_state
    result = self.graphql(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/client.py", line 464, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]```

Update: Brett opened a support case via email, we are getting help through that channel. I'll post the resolution here when we have one.

It turned out to be Task Concurrency. Deleting the references to Task Concurrency in our code was not enough, finally fixing the issue for good required deleting the limits using the Prefect API.