https://prefect.io logo
Channels
Powered by
f

Frank Hardisty

02/01/2023, 12:24 AM
Hello, Frank from Replica here. Yesterday we were having an issue with Flow Runs failing with `API_ERROR`s. Yesterday we were having similar issues, my colleague @Brett Naul reached out and the issue seemed resolved, however it's happening again. Can anyone advise? Details in thread. Thanks in advance.
✅ 1
Message logged:
Copy code
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/client.py", line 1618, in set_task_run_state
    result = self.graphql(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/client.py", line 464, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]
Update: Brett opened a support case via email, we are getting help through that channel. I'll post the resolution here when we have one.
It turned out to be Task Concurrency. Deleting the references to Task Concurrency in our code was not enough, finally fixing the issue for good required deleting the limits using the Prefect API.
Powered by