Hey guys, I’m making a database query as a snowflake task inside a Prefect Cloud flow. The output of this query takes up memory size that’s larger than 1 MB so when the Task tries to call

set_task_result()

it fails with error message:

'State payload is too large.'

which makes sense. My Prefect infrastructure is sitting on a KubernetesJobEnvironment. My understanding is that to store this large result correctly I should follow the documentation here: https://docs.prefect.io/core/concepts/results.html#result-objects Are there any additional considerations I should take into account with setting up correct result storage with this infrastructure?

If you want to, you may disable storing results for that task by adding this to your task decorator:

@task(checkpoint=False)

For the SnowflakeQuery task you could set it when you initialize the task:

query = SnowflakeQuery(checkpoint=False)

with Flow("yourflow") as flow:
    query(*run_args)

if I disable results for the snowflake query can I still pass the data that it yields downstream in the flow?

No worries! The checkpointing is primarily relevant for restarts from the UI, great to hear your issue is resolved! 🙌

that’s excellent. is the Apollo API 5 MB limit something I would have to consider when I scale up?

if you are on Prefect Cloud, you don't have to worry about Apollo. And the payload limits would only be relevant if you send overly large API requests, but what you do here is passing data in-memory so this is different

oh wonderful and since im on k8s i should just be able to scale up the pod mem size if it gets to that point