hey guys, I'm having a bit of a weird issue where a specific task is causing this error in Prefect:

Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State payload is too large.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1518, in set_task_run_state
    result = self.graphql(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State payload is too large.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

It appears to me that I'm managing a JSON payload that Prefect considers "too large" however from what I see, the payload I'm sending is only on the scale of ~100KB which to me intuitively isn't that large. Is there some limitation in Prefect that I'm accidentally breaching or is there something else I should be investigating in my code that could be causing this

Hey kevin, we’re chatting about this internally. I’m not sure why you’d be running into payload limits here, but it’d be good to inspect the size of the payload being sent in the client to confirm the size (or briefly explain how you inspected to already).

i was just eyeballing input size based on the code transforms and input data we're providing, i can get you a more concrete measurement if that helps

Also, I added that comment while doing some refactoring in the client but it looks like that code does indeed execute 🙂 only 400 status codes are raised in

_send_request

im glad to hear that im not going insane haha

Yeah if you could display the size of the serialized state at https://github.com/PrefectHQ/prefect/blob/master/src/prefect/client/client.py#L1915 that’d be helpful

our task runs post JSON to an API outside of prefect that writes the data so we aren't explicitly writing any data with Prefect

Are you returning a value from the task? Doing any particularly wild amounts of downstream or upstream edges?

we're just returning a UUID from the task, no large amount of edges downstream or upstream, we're taking in two inputs upstream and one output input downstream

Weird. Can you share a minimal reproducible example?

what's the best way for me to present that to you?

Preferably a short flow script I can run to reproduce the issue.

our flows use a lot of internal libraries to do things, I can psuedo code the gist of what we're attempting if that works?

That may help, but what we’re trying to do here is narrow down where this large payload is coming from and pseudocode likely wont get us there.

oh okay got it

You can also put something like this in the client (where I linked to)

state_size = sys.getsizeof(json.dumps(serialized_state))
if state_size > 1000000:  # 1 mb
    breakpoint()

oh okay

Or just step up into this frame from a debugger when that error occurs and print the serialized state.

yea lemme spin my wheels for a second and get that goin

You can run the flow without an agent (but reporting to Cloud) using

prefect run --execute

so you can use a debugger

that is super neat and i did not know that thanks

There’s a blurb here under “Agentless execution” if that’s helpful https://docs.prefect.io/orchestration/flow-runs/creation.html#cli

hey sorry for the delay, i was wrangling with some isues w/ our prod and staging envorinments

Glad you found the problem!

thank you for your patience 🙂