Eddie Atkinson
04/12/2022, 2:35 AMECSRun
flow using the LocalDaskExecutor
with 30GB of RAM. For large jobs this flow OOMs and gets killed by the Prefect scheduler. My question is this: If I set up a Dask cluster to run these jobs would it gracefully handle memory issues?
That is to say if I had 30GB of RAM in the cluster and a job that required 50GB would Dask OOM or would it simply run slower? Do I need to modify my code to use Dask dataframes or is there some smarts here I’m not quite across?Kevin Kho
04/12/2022, 2:38 AMEddie Atkinson
04/12/2022, 3:34 AM