Hi Prefect community, I am running into a weird memory issue on my deployment. I m using Prefect Core + Server with a K8s agent and a temporary dask executor. My DAG contains a lot of tasks (because of some large mapping). The dask scheduler memory grows from few 100Mb to several Gb before being killed. I can't really explain why s that, but it might be due to my limited Dask distributed understanding. I also have checked the issue on Github, and I found that https://github.com/PrefectHQ/prefect/issues/4085 I m personally using an Azure Blob as storage. Could it be the same issue? To test that out, I wanted to switch to a Module storage, but for a reason that I dont understand, it does not spin up the temporary dask clsuter and runs everything on the K8s job. I would love to hear any suggestions. Thanks

Did you try to adjust the memory limit on the dask scheduler pod? This is something we had to do when we started with perfect to ensure it scale properly

Hi @davzucky, the scheduler runs on a node with 12G of memory, and get OOMkilled once it reach 12G. Is that normal for a dask scheduler to reach such a high RAM usage?

After we had problem with dask worker consuming all memory and not been getting killed. This is because dark worker keep in memory the results. You need to be careful at that as well

Yeah I read about such a thing, but in my case, the scheduler get killed, not the workers

How many task do you run,? I had a mapped task of over 120k entries and didn't get scheduler problem

It might be ~few k entries. I can't really share a sample of my code but I would have to try to find a dummy flow to replicate this issue.

How many input did you talk has?

Depends on the task, but let s say something like ~ 5 inputs. But I am not sure why this matters? I would expect the dask scheduler memory to stay relatively constant, and the worker memories to grow. Do you know what the scheduler is keeping in memory beside the DAG?

I think they scheduler is keeping the serialized tasks as well. However I'm not sure of that one but this is something I was going to investigate