Jan Rouš01/14/2021, 3:27 AM
the implicit in-memory caching results in the ETL eventually getting oom-killed 😕 I'm using checkpointing and some of the tasks explicitly write their results to disk where they are picked up by subsequent stages so the in-memory caching is not strictly necessary. I have not really found a way how I could tell prefect to not bother with this or at least, not eat all available memory while doing so. Any pointers for how I could solve this problems would be really helpful!
Jan Rouš01/14/2021, 5:49 PM
and thus bypass the in-memory caching?
Jan Rouš01/14/2021, 6:04 PM
@task def A(x): ... @task def B(y): ... @task def Aggregate(x): y = A.run(x) return B.run(y)
Jan Rouš01/15/2021, 3:49 AM