Hi All,
I've a question regarding in memory caching in Prefect, if I read this doc, it says caching is enabled by default, so let's say if I run the same task multiple times from a loop inside a flow, result of all task runs will be cached ?
https://docs.prefect.io/2.11.3/concepts/results/#caching-of-results-in-memory
Tanishq Hooda
08/10/2023, 5:29 AM
Copy code
from prefect import flow, task
from prefect import get_run_logger
@task
def huge_data(i):
return "Return Huge data!"
@flow(name="test_flow")
def test_flow():
for i in range(5):
huge_data(i)
if __name__ == "__main__":
test_flow()
in this case huge_data runs 5 times, and let's say the function returns 100 MB of data, will I have a cache of 100*5 = 500 MB at the end of it?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
