Bradley Hurley

10/26/2022, 9:53 PM
Hi Prefect Experts - Im running Prefect 1, but running into an issue that Im having a hard time tracking down. We have Prefect deployed on Kuberneters and are using Dask Executors. The general pattern is: users submits a flow agents creates a job pod job pod creates a dask cluster (minimum of 1 scheduler and 2 workers) tasks run on the dask workers The job pod is consuming memory until it crashes.
on the job pod we can see the prefect process is running
prefect execute flow-run
until all the memory on the physical node is consumed.
I didn’t anticipate much happening on the job pod since we were executing tasks via dask
From the agent
prefect version

Kalise Richmond

10/26/2022, 10:34 PM
Hi Bradley, you might find this discourse article useful in understanding how prefect sends work to dask and handles memory.

Bradley Hurley

10/26/2022, 10:58 PM
Thanks! I will review.
We are certainly using mapped tasks, but we don’t return much data. Almost everything is a pointer to data on S3.