Hi. I have a mapped task that pulls data from an API and passes it to a reduce task that saves the data to S3. Some of the API calls may fail and that is OK. I am using a

FilterTask

as explained here to allow the reduce task to save the data. It works ok with small results, but when I try it with the real API output, I am getting

State payload is too large

I tried setting

result=None

for this task but that did not fix it. Any suggestions?

Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State payload is too large.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1917, in set_task_run_state
    result = self.graphql(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 569, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State payload is too large.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

I also tried setting the trigger on the reduce task to

trigger=any_successful

and it worked but I wonder why the filter task is not working. Thanks!

The state payload has a limit of 1MB. Do you have long messages with your states?

@Pedro Machado can you share how your flow looks like? hard to tell anything right now without seeing the flow structure, especially results.