https://prefect.io logo
Title
a

Andrew Lawlor

01/26/2023, 2:43 PM
seeing a lot of api errors today. have seen the following error message in 4 different flows overnight. have never seen it before. any idea on how to fix?
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]) Traceback (most recent call last):   File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers     state = self.client.set_task_run_state(   File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1622, in set_task_run_state     result = self.graphql(   File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 465, in graphql     raise ClientError(result["errors"]) prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]
this is prefect 1.x
c

Christopher Boyd

01/26/2023, 3:05 PM
HI Andrew, Do you have any time stamps you can correlate, so we can check?
👀 1
a

Andrew Lawlor

01/26/2023, 3:23 PM
05:26:20
thats eastern time
c

Christopher Boyd

01/26/2023, 3:25 PM
Can you share any task / flow_run id’s as well to correlate? Is that just teh latest error that you saw? What about the others? Also, do you have any issues with task / flow concurrency that might be related to this, or any other specifics ?
a

Andrew Lawlor

01/26/2023, 3:47 PM
that task run was 6c2adee0-3ace-41ac-b6f0-6b8317eb9ddf it was the latest one we saw. we had flows working for a long time, and then all of a sudden several of them started failing. there seem to be issues with our other returns from the api. my coworker is talking to support via email. i also saw it at 22:32:24 on task ac4952c5-eb3a-49d1-aafc-69f36032ed26 with a different flow
j

Jacob Blanco

02/14/2023, 12:10 AM
Did you get any response on this? We are seeing the exact same errors now more and more.
c

Christopher Boyd

02/14/2023, 2:38 PM
Are you on Prefect 1?
j

Jacob Blanco

02/14/2023, 2:44 PM
Yes
Here are vague notions of things that might be related: • The flows where this is happening are older • They have a bunch of mapping (some as many as >50 per run per task for about 6-8 tasks) • They have some form of concurrency (task or flow)
c

Christopher Boyd

02/14/2023, 2:46 PM
We are looking into it
j

Jacob Blanco

02/14/2023, 2:47 PM
Thanks
If I pull the tasks that are in running state with a start time of 3 days ago (which there should be very few) I get 1000 records back and I assume that’s a limit on the API. That doesn’t seem normal. I am seeing Running tasks since before 2022-06. Looking at it now, I guess the
task_run_state
query can return any task run state even past ones. NVM.
@Salim Doost @Jezreel Zamora Waiting on response here from Christopher
👀 2
c

Christopher Boyd

02/14/2023, 6:26 PM
We are seeing some performance delays on the concurrency index, it’s being worked on now
j

Jacob Blanco

02/15/2023, 9:36 AM
Thank you!