seeing a lot of api errors today.
have seen the f...
# ask-communitya
Andrew Lawlor
01/26/2023, 2:43 PMseeing a lot of api errors today.
have seen the following error message in 4 different flows overnight. have never seen it before. any idea on how to fix?
Copy code
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]) Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers state = self.client.set_task_run_state( File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1622, in set_task_run_state result = self.graphql( File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 465, in graphql raise ClientError(result["errors"]) prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]Andrew Lawlor
01/26/2023, 2:46 PMthis is prefect 1.x
c
Christopher Boyd
01/26/2023, 3:05 PMHI Andrew,
Do you have any time stamps you can correlate, so we can check?
👀 1
a
Andrew Lawlor
01/26/2023, 3:23 PM052620
Andrew Lawlor
01/26/2023, 3:23 PMthats eastern time
c
Christopher Boyd
01/26/2023, 3:25 PMCan you share any task / flow_run id’s as well to correlate?
Is that just teh latest error that you saw? What about the others?
Also, do you have any issues with task / flow concurrency that might be related to this, or any other specifics ?
a
Andrew Lawlor
01/26/2023, 3:47 PMthat task run was 6c2adee0-3ace-41ac-b6f0-6b8317eb9ddf
it was the latest one we saw. we had flows working for a long time, and then all of a sudden several of them started failing. there seem to be issues with our other returns from the api. my coworker is talking to support via email.
i also saw it at 223224
on task ac4952c5-eb3a-49d1-aafc-69f36032ed26 with a different flow
j
Jacob Blanco
02/14/2023, 12:10 AMDid you get any response on this? We are seeing the exact same errors now more and more.
c
Christopher Boyd
02/14/2023, 2:38 PMAre you on Prefect 1?
j
Jacob Blanco
02/14/2023, 2:44 PMYes
Jacob Blanco
02/14/2023, 2:46 PMHere are vague notions of things that might be related:
• The flows where this is happening are older
• They have a bunch of mapping (some as many as >50 per run per task for about 6-8 tasks)
• They have some form of concurrency (task or flow)
c
Christopher Boyd
02/14/2023, 2:46 PMWe are looking into it
j
Jacob Blanco
02/14/2023, 2:47 PMThanks
Jacob Blanco
02/14/2023, 3:05 PMtask_run_stateJacob Blanco
02/14/2023, 6:14 PM@Salim Doost @Jezreel Zamora Waiting on response here from Christopher
👀 2
c
Christopher Boyd
02/14/2023, 6:26 PMWe are seeing some performance delays on the concurrency index, it’s being worked on now
j
Jacob Blanco
02/15/2023, 9:36 AMThank you!
4 Views