https://prefect.io logo
Title
b

Bradley Hurley

10/26/2022, 9:53 PM
Hi Prefect Experts - Im running Prefect 1, but running into an issue that Im having a hard time tracking down. We have Prefect deployed on Kuberneters and are using Dask Executors. The general pattern is: users submits a flow agents creates a job pod job pod creates a dask cluster (minimum of 1 scheduler and 2 workers) tasks run on the dask workers The job pod is consuming memory until it crashes.
Running
htop
on the job pod we can see the prefect process is running
prefect execute flow-run
until all the memory on the physical node is consumed.
I didn’t anticipate much happening on the job pod since we were executing tasks via dask
From the agent
prefect version
1.4.0
k

Kalise Richmond

10/26/2022, 10:34 PM
Hi Bradley, you might find this discourse article useful in understanding how prefect sends work to dask and handles memory.
b

Bradley Hurley

10/26/2022, 10:58 PM
Thanks! I will review.
We are certainly using mapped tasks, but we don’t return much data. Almost everything is a pointer to data on S3.