Jan Rouš
01/14/2021, 3:27 AMLocalExecutor
the implicit in-memory caching results in the ETL eventually getting oom-killed 😕 I'm using checkpointing and some of the tasks explicitly write their results to disk where they are picked up by subsequent stages so the in-memory caching is not strictly necessary. I have not really found a way how I could tell prefect to not bother with this or at least, not eat all available memory while doing so. Any pointers for how I could solve this problems would be really helpful!Zanie
Jan Rouš
01/14/2021, 5:49 PMresult=GCSResult(...)
and thus bypass the in-memory caching?Zanie
Jan Rouš
01/14/2021, 6:04 PM@task
def A(x):
...
@task
def B(y):
...
@task
def Aggregate(x):
y = A.run(x)
return B.run(y)
Zanie
Jan Rouš
01/15/2021, 3:49 AM