Hi Prefect Experts Im running Prefect 1 but running into an

Question

Hi Prefect Experts Im running Prefect 1 but running into an issue that Im having a hard time tracking down We have Prefect deployed on Kuberneters and are using Dask Executors The general pattern is users submits a flow agents creates a job pod job pod creates a dask cluster minimum of 1 scheduler and 2 workers tasks run on the dask workers The job pod is consuming memory until it crashes

Bradley Hurley · Answer

Running `htop` on the job pod we can see the prefect process is running `prefect execute flow run` until all the memory on the physical node is consumed

Bradley Hurley · Answer

I didn t anticipate much happening on the job pod since we were executing tasks via dask

Bradley Hurley · Answer

From the agent ```prefect version 1 4 0```

Kalise Richmond · Answer

Hi Bradley you might find <https discourse prefect io t how does prefect send work to a daskexecutor and handle memory 468|this> discourse article useful in understanding how prefect sends work to dask and handles memory

Bradley Hurley · Answer

Thanks I will review

Bradley Hurley · Answer

We are certainly using mapped tasks but we don t return much data Almost everything is a pointer to data on S3