Normally our flows run successfully, without any problems. However, we encountered this error for two flow runs:

Failed to set task state with error: ClientError([{'message': 'Unable to complete operation. An internal API error occurred.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Unable to complete operation. An internal API error occurred.'}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1598, in set_task_run_state
    result = self.graphql(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 473, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': 'Unable to complete operation. An internal API error occurred.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Unable to complete operation. An internal API error occurred.'}}}]

In the same timespan, we had 5 flows running, 3 succeeded, only the 2 failed with the above error. The flows run every hour, in the previous hour and next hour everything was running fine.

start_time                          end_time                            state   state_message
2022-04-08T02:00:13.555177+00:00	2022-04-08T02:14:19.782245+00:00	Failed	Some reference tasks failed.
2022-04-08T02:00:13.558573+00:00	2022-04-08T02:14:23.027744+00:00	Success	All reference tasks succeeded.
2022-04-08T02:00:13.849767+00:00	2022-04-08T02:14:25.492537+00:00	Success	All reference tasks succeeded.
2022-04-08T02:00:13.929466+00:00	2022-04-08T02:14:23.787185+00:00	Failed	Some reference tasks failed.
2022-04-08T02:00:14.427653+00:00	2022-04-08T02:21:56.436942+00:00	Success	All reference tasks succeeded.

The internal server error is a generic message from an API call that failed. It could be that: • your payload was too large • your container died Could you send the code of this failing flow? Are you using some large dictionaries as Parameter task values, perhaps? Hard to say because the error is so generic

@Anna Geller Could actually be large payloads, we're trying to look into it further, thanks. The code is unfortunately part of our IP, so cannot really share it. Just a suggestion; Would it be possible to add a check in Prefect, checking the size of the results/parameters and output a warning with a more informative message?

I don't think this will tackle the root cause of the problem, though. Even if you get the message, your flow will still fail. A more constructive approach would be to redesign your workflow so that instead of sending large payloads via a Parameter task, you instead send only a reference to some stateful object that contains the actual data, e.g. reference to S3 location, key in some key-value store, etc