Sylvain Hazard

    Sylvain Hazard

    10 months ago
    Hi ! I have some questions on memory usage in Prefect. We're running our flow using Kubernetes Run with a LocalDaskExecutor. The flow maps a bunch of tasks, with a total of about 50K tasks for it to finish. What we see is that the memory usage of the runner pod steadily increases for all the duration of the flow. Whenever it reaches its predefined memory limit, we see a sudden drop in usage but the flow keeps running properly. Is it something that is to be expected from a flow with that many tasks ? Is there anything we could do to prevent using as much memory ?
    Anna Geller

    Anna Geller

    10 months ago
    @Sylvain Hazard
    LocalDaskExecutor
    can only parallelize work within a single machine, in your use case within a single pod. The easiest way to distribute this work would be offloading it to a distributed Dask cluster. When you use Kubernetes, you could follow this tutorial to set up a temporary Dask cluster. This would likely help with the memory usage, as Dask scheduler would manage this.
    Sylvain Hazard

    Sylvain Hazard

    10 months ago
    I expected the LocalDaskExecutor to be able to run some kind of garbage collecting whenever tasks end or something of that nature. Guess I'm gonna have to make the switch to a proper Dask Executor ! Thanks a bunch !
    Kevin Kho

    Kevin Kho

    10 months ago
    I think you can try explicitly using the Python garbage collection to delete stuff
    del something; gc.collect()
    . If you are passing large objects around, you can also try saving them somewhere, passing the location, and then loading it in a downstream task.
    Evan Curtin

    Evan Curtin

    10 months ago
    take a look at this

    https://www.youtube.com/watch?v=nwR6iGR0mb0

    Kevin Kho

    Kevin Kho

    10 months ago
    Have you tried this for LocalDask @Evan Curtin? I am curious if it would work as well
    Evan Curtin

    Evan Curtin

    10 months ago
    not via prefect but it helped me a lot using a dask LocalCluster
    I imagine that’s what is going on under the hood, no?
    Kevin Kho

    Kevin Kho

    10 months ago
    That should work then I guess. It's more like a multiprocessing pool (does not use
    distributed
    )