Is the a parameter for flow/tasks to not cache/store anything locally?
n
Nate
04/29/2024, 3:39 PM
hi @Chris - if you do not configure results, then by default the sdk will not persist / cache results locally
does that answer your question?
c
Chris
04/29/2024, 3:40 PM
I think so
I started doing
@flow(cache_result_in_memory=False)
and my memory usage seems much better. Is there another config for not storing/caching?
n
Nate
04/29/2024, 3:41 PM
to clarify, you're mostly wondering about things stored in memory, not on disk? and yeah
cache_result_in_memory=False
should avoid taking up extra space in memory
c
Chris
04/29/2024, 3:42 PM
Yes just memory, no issue with disk atm.
n
Nate
04/29/2024, 3:45 PM
cool,
cache_result_in_memory
is a lever you have there. is there a specific problem you're still hitting?
other general tips would be:
⢠pass references to really large data
⢠use
prefect.annotations.quote(really_big_input_param)
to avoid inspection of some massive task parameter (if its just static data, and not futures that need to be resolved, since quote will also skip that)
š 1
c
Chris
04/29/2024, 3:52 PM
Na just wanted to make sure that was it. Not super clear to me what happens under the hood at times. Thanks for the help
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.