kevin

    kevin

    10 months ago
    hey guys, I'm having a bit of a weird issue where a specific task is causing this error in Prefect:
    Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State payload is too large.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
        state = self.client.set_task_run_state(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1518, in set_task_run_state
        result = self.graphql(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
        raise ClientError(result["errors"])
    prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State payload is too large.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
    It appears to me that I'm managing a JSON payload that Prefect considers "too large" however from what I see, the payload I'm sending is only on the scale of ~100KB which to me intuitively isn't that large. Is there some limitation in Prefect that I'm accidentally breaching or is there something else I should be investigating in my code that could be causing this
    As an additional thing: I've tracked this code to this line https://github.com/PrefectHQ/prefect/blob/0464042e574c8187d0c344e52a4c39727f378867/src/prefect/client/client.py#L569 which interestingly enough according to the comments should never execute? Maybe I'm reading it wrong
    Michael Adkins

    Michael Adkins

    10 months ago
    Hey kevin, we’re chatting about this internally. I’m not sure why you’d be running into payload limits here, but it’d be good to inspect the size of the payload being sent in the client to confirm the size (or briefly explain how you inspected to already).
    kevin

    kevin

    10 months ago
    i was just eyeballing input size based on the code transforms and input data we're providing, i can get you a more concrete measurement if that helps
    Michael Adkins

    Michael Adkins

    10 months ago
    Also, I added that comment while doing some refactoring in the client but it looks like that code does indeed execute 🙂 only 400 status codes are raised in
    _send_request
    kevin

    kevin

    10 months ago
    im glad to hear that im not going insane haha
    Michael Adkins

    Michael Adkins

    10 months ago
    Yeah if you could display the size of the serialized state at https://github.com/PrefectHQ/prefect/blob/master/src/prefect/client/client.py#L1915 that’d be helpful
    We’ve got a 5MB payload limit in Apollo, but a 1MB limit for state payloads specifically.
    Are you using a
    PrefectResult
    to store your task results in the database?
    kevin

    kevin

    10 months ago
    our task runs post JSON to an API outside of prefect that writes the data so we aren't explicitly writing any data with Prefect
    Michael Adkins

    Michael Adkins

    10 months ago
    Are you returning a value from the task? Doing any particularly wild amounts of downstream or upstream edges?
    kevin

    kevin

    10 months ago
    we're just returning a UUID from the task, no large amount of edges downstream or upstream, we're taking in two inputs upstream and one output input downstream
    also I had a bit of a struggle accurately capturing the payload size so I just opened chrome inspector and reran the flow and tracked the size of every post request
    it never exceeds 40KB
    Michael Adkins

    Michael Adkins

    10 months ago
    Weird. Can you share a minimal reproducible example?
    kevin

    kevin

    10 months ago
    what's the best way for me to present that to you?
    Michael Adkins

    Michael Adkins

    10 months ago
    Preferably a short flow script I can run to reproduce the issue.
    kevin

    kevin

    10 months ago
    our flows use a lot of internal libraries to do things, I can psuedo code the gist of what we're attempting if that works?
    Michael Adkins

    Michael Adkins

    10 months ago
    That may help, but what we’re trying to do here is narrow down where this large payload is coming from and pseudocode likely wont get us there.
    kevin

    kevin

    10 months ago
    oh okay got it
    Michael Adkins

    Michael Adkins

    10 months ago
    You can also put something like this in the client (where I linked to)
    state_size = sys.getsizeof(json.dumps(serialized_state))
    if state_size > 1000000:  # 1 mb
        breakpoint()
    kevin

    kevin

    10 months ago
    oh okay
    Michael Adkins

    Michael Adkins

    10 months ago
    Or just step up into this frame from a debugger when that error occurs and print the serialized state.
    kevin

    kevin

    10 months ago
    yea lemme spin my wheels for a second and get that goin
    Michael Adkins

    Michael Adkins

    10 months ago
    You can run the flow without an agent (but reporting to Cloud) using
    prefect run --execute
    so you can use a debugger
    kevin

    kevin

    10 months ago
    that is super neat and i did not know that thanks
    Michael Adkins

    Michael Adkins

    10 months ago
    There’s a blurb here under “Agentless execution” if that’s helpful https://docs.prefect.io/orchestration/flow-runs/creation.html#cli
    kevin

    kevin

    10 months ago
    hey sorry for the delay, i was wrangling with some isues w/ our prod and staging envorinments
    the byte size of the payload that I ran it with was
    119876
    which is about a tenth of the max size i believe
    okay I figured it out, it turns out it was entirely on my end. Our production data blob is 10x our staging and testing ones i'm actually so sorry for this kerfuffle
    Michael Adkins

    Michael Adkins

    10 months ago
    Glad you found the problem!
    kevin

    kevin

    10 months ago
    thank you for your patience 🙂