https://prefect.io logo
#prefect-server
Title
# prefect-server
t

Thomas Pedersen

04/08/2022, 9:01 AM
Normally our flows run successfully, without any problems. However, we encountered this error for two flow runs:
Copy code
Failed to set task state with error: ClientError([{'message': 'Unable to complete operation. An internal API error occurred.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Unable to complete operation. An internal API error occurred.'}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1598, in set_task_run_state
    result = self.graphql(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 473, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': 'Unable to complete operation. An internal API error occurred.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Unable to complete operation. An internal API error occurred.'}}}]
In the same timespan, we had 5 flows running, 3 succeeded, only the 2 failed with the above error. The flows run every hour, in the previous hour and next hour everything was running fine.
Copy code
start_time                          end_time                            state   state_message
2022-04-08T02:00:13.555177+00:00	2022-04-08T02:14:19.782245+00:00	Failed	Some reference tasks failed.
2022-04-08T02:00:13.558573+00:00	2022-04-08T02:14:23.027744+00:00	Success	All reference tasks succeeded.
2022-04-08T02:00:13.849767+00:00	2022-04-08T02:14:25.492537+00:00	Success	All reference tasks succeeded.
2022-04-08T02:00:13.929466+00:00	2022-04-08T02:14:23.787185+00:00	Failed	Some reference tasks failed.
2022-04-08T02:00:14.427653+00:00	2022-04-08T02:21:56.436942+00:00	Success	All reference tasks succeeded.
Using Prefect version 1.0.0 and DockerRun with image based on prefecthq/prefect:1.0.0-python3.9 ... might be trying to upgrade prefect in a week or two to see if it prevents it from occurring again.
a

Anna Geller

04/08/2022, 10:08 AM
The internal server error is a generic message from an API call that failed. It could be that: • your payload was too large • your container died Could you send the code of this failing flow? Are you using some large dictionaries as Parameter task values, perhaps? Hard to say because the error is so generic
t

Thomas Pedersen

04/08/2022, 10:32 AM
@Anna Geller Could actually be large payloads, we're trying to look into it further, thanks. The code is unfortunately part of our IP, so cannot really share it. Just a suggestion; Would it be possible to add a check in Prefect, checking the size of the results/parameters and output a warning with a more informative message?
a

Anna Geller

04/08/2022, 10:44 AM
I don't think this will tackle the root cause of the problem, though. Even if you get the message, your flow will still fail. A more constructive approach would be to redesign your workflow so that instead of sending large payloads via a Parameter task, you instead send only a reference to some stateful object that contains the actual data, e.g. reference to S3 location, key in some key-value store, etc
52 Views