https://prefect.io logo
#prefect-community
Title
# prefect-community
k

kevin

04/08/2022, 7:36 AM
Hey guys, I’m making a database query as a snowflake task inside a Prefect Cloud flow. The output of this query takes up memory size that’s larger than 1 MB so when the Task tries to call
set_task_result()
it fails with error message:
'State payload is too large.'
which makes sense. My Prefect infrastructure is sitting on a KubernetesJobEnvironment. My understanding is that to store this large result correctly I should follow the documentation here: https://docs.prefect.io/core/concepts/results.html#result-objects Are there any additional considerations I should take into account with setting up correct result storage with this infrastructure?
a

Anna Geller

04/08/2022, 9:34 AM
If you want to, you may disable storing results for that task by adding this to your task decorator:
Copy code
@task(checkpoint=False)
For the SnowflakeQuery task you could set it when you initialize the task:
Copy code
query = SnowflakeQuery(checkpoint=False)

with Flow("yourflow") as flow:
    query(*run_args)
k

kevin

04/08/2022, 3:15 PM
if I disable results for the snowflake query can I still pass the data that it yields downstream in the flow?
sorry, my above question was a result of misunderstanding what checkpointing does. Setting that flag to False did resolve my issue. Thanks for helping me out 🙂
a

Anna Geller

04/08/2022, 6:04 PM
No worries! The checkpointing is primarily relevant for restarts from the UI, great to hear your issue is resolved! 🙌
k

kevin

04/08/2022, 6:05 PM
that’s excellent. is the Apollo API 5 MB limit something I would have to consider when I scale up?
a

Anna Geller

04/08/2022, 6:20 PM
if you are on Prefect Cloud, you don't have to worry about Apollo. And the payload limits would only be relevant if you send overly large API requests, but what you do here is passing data in-memory so this is different
k

kevin

04/08/2022, 6:21 PM
oh wonderful and since im on k8s i should just be able to scale up the pod mem size if it gets to that point
thank you for clearing up my confusions
🙌 1
4 Views