https://prefect.io
Join Slack
seeing a lot of api errors today. have seen the following error message in 4 different flows overni...
# ask-community
a
seeing a lot of api errors today. have seen the following error message in 4 different flows overnight. have never seen it before. any idea on how to fix?
Copy code
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]) Traceback (most recent call last):   File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers     state = self.client.set_task_run_state(   File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1622, in set_task_run_state     result = self.graphql(   File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 465, in graphql     raise ClientError(result["errors"]) prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]
this is prefect 1.x
c
HI Andrew, Do you have any time stamps you can correlate, so we can check?
👀 1
a
052620
thats eastern time
c
Can you share any task / flow_run id’s as well to correlate? Is that just teh latest error that you saw? What about the others? Also, do you have any issues with task / flow concurrency that might be related to this, or any other specifics ?
a
that task run was 6c2adee0-3ace-41ac-b6f0-6b8317eb9ddf it was the latest one we saw. we had flows working for a long time, and then all of a sudden several of them started failing. there seem to be issues with our other returns from the api. my coworker is talking to support via email. i also saw it at 223224 on task ac4952c5-eb3a-49d1-aafc-69f36032ed26 with a different flow
j
Did you get any response on this? We are seeing the exact same errors now more and more.
c
Are you on Prefect 1?
j
Yes
Here are vague notions of things that might be related: • The flows where this is happening are older • They have a bunch of mapping (some as many as >50 per run per task for about 6-8 tasks) • They have some form of concurrency (task or flow)
c
We are looking into it
j
Thanks
If I pull the tasks that are in running state with a start time of 3 days ago (which there should be very few) I get 1000 records back and I assume that’s a limit on the API. That doesn’t seem normal. I am seeing Running tasks since before 2022-06. Looking at it now, I guess the 
task_run_state
query can return any task run state even past ones. NVM.
@Salim Doost @Jezreel Zamora Waiting on response here from Christopher
👀 2
c
We are seeing some performance delays on the concurrency index, it’s being worked on now
j
Thank you!
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by