seeing a lot of api errors today. have seen the following error message in 4 different flows overnight. have never seen it before. any idea on how to fix?

Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]) Traceback (most recent call last):   File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers     state = self.client.set_task_run_state(   File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1622, in set_task_run_state     result = self.graphql(   File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 465, in graphql     raise ClientError(result["errors"]) prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]

HI Andrew, Do you have any time stamps you can correlate, so we can check?

05:26:20

Can you share any task / flow_run id’s as well to correlate? Is that just teh latest error that you saw? What about the others? Also, do you have any issues with task / flow concurrency that might be related to this, or any other specifics ?

that task run was 6c2adee0-3ace-41ac-b6f0-6b8317eb9ddf it was the latest one we saw. we had flows working for a long time, and then all of a sudden several of them started failing. there seem to be issues with our other returns from the api. my coworker is talking to support via email. i also saw it at 22:32:24 on task ac4952c5-eb3a-49d1-aafc-69f36032ed26 with a different flow

Did you get any response on this? We are seeing the exact same errors now more and more.

Are you on Prefect 1?

Yes

We are looking into it

Thanks

We are seeing some performance delays on the concurrency index, it’s being worked on now

Thank you!