Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
f
Frank Hardisty
02/01/2023, 12:24 AMHello, Frank from Replica here. Yesterday we were having an issue with Flow Runs failing with `API_ERROR`s. Yesterday we were having similar issues, my colleague @Brett Naul reached out and the issue seemed resolved, however it's happening again. Can anyone advise? Details in thread. Thanks in advance.
✅ 1
Message logged:
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}])
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/usr/local/lib/python3.10/site-packages/prefect/client/client.py", line 1618, in set_task_run_state
result = self.graphql(
File "/usr/local/lib/python3.10/site-packages/prefect/client/client.py", line 464, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]Update: Brett opened a support case via email, we are getting help through that channel. I'll post the resolution here when we have one.
It turned out to be Task Concurrency. Deleting the references to Task Concurrency in our code was not enough, finally fixing the issue for good required deleting the limits using the Prefect API.