We are seeing quite a lot intermittent of API erro...
# prefect-communityf
Florian Kühnlenz
12/17/2022, 7:53 PMWe are seeing quite a lot intermittent of API errors and consequent flow run failures in cloud v1. Anyone else?
a
Anna Geller
12/17/2022, 8:00 PM
thanks for reporting Florian, I forwarded to the team
f
Florian Kühnlenz
12/17/2022, 8:05 PMThanks. Example run-id 432371db-b3eb-400e-9647-e7352c25663c
gratitude thank you 1
a
Anna Geller
12/17/2022, 8:28 PM
are you sure this is the right ID? we don't see any errors with that ID
Anna Geller
12/17/2022, 9:31 PM
we cross-checked with the team and nothing stands out in particular, unfortunately. if the situation continues, feel free to add more details (logs, more flow run IDs) - atm things look stable. Sorry for not being more helpful here
🙏 1
👍 2
f
Florian Kühnlenz
12/18/2022, 5:41 AMHere are some another IDs bc3f2411-bca8-47af-8125-a90d12ed946b, bc3f2411-bca8-47af-8125-a90d12ed946b and the first one again 432371db-b3eb-400e-9647-e7352c25663c.
We got about 5 more over night. They all contain the same error:
Copy code
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}])
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1922, in set_task_run_state
result = self.graphql(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 570, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]Florian Kühnlenz
12/18/2022, 5:42 AM@Anna Geller to me it seams some of them got rescheduled but only the tasks that did not run prior to the API error. Which does not work if no intermediate results are saved.
a
Anna Geller
12/18/2022, 8:56 PM
we investigated and we see errors related to version lock:
Copy code
VERSION LOCK: State update failed for flow run ID bc3f2411-bca8-47af-8125-a90d12ed946b: provided version 1 but current version is 2.f
Florian Kühnlenz
12/19/2022, 5:38 AMNo we did not try. We also did not see any new errors of this kind yesterday. Is there a way to enable version locking globally?
a
Anna Geller
12/19/2022, 1:00 PM
You could write a script to list flows and enable it per each flow programmatically via GraphQL API but no single endpoint to do that globally
Anna Geller
12/19/2022, 1:01 PM
We have some similar GraphQL query examples on Discourse IIRC
f
Florian Kühnlenz
12/19/2022, 2:46 PMThe reason I am asking is, because this is the second time that I get this Feature recommended, for a totally different problem, and never are any downsides mentioned. So I wonder why this is not turned on globally.
a
Anna Geller
12/19/2022, 3:39 PM
version locking enforces that your flow runs only once and that's not always desirable - we have users who want to run flows any time for any reason, so having that as a global default is, as with anything in engineering, a tradeoff
Anna Geller
12/19/2022, 3:47 PM
I got more concrete reasons from our experts
1. Version locking is not free, there is a cost imposed with respect to performance/latency
2. Version locking can have unexpected coordination effects like flow runs crashing / not recovering properly due to version lock errors and those errors can sometimes be hard to understand/troubleshoot if enabled for everything, so it's better to enabled it per flow when needed
🙌 1
f
Florian Kühnlenz
12/20/2022, 5:10 AMIf it limits the flow to run only once per version it is completely useless in both cases it was suggested to us. Thanks for clearing that up.
2 Views