Thread
#prefect-community
    Max Kureykin

    Max Kureykin

    1 year ago
    Hi guys! The question is about prefect/dask combo. Is there any way to limit dask scheduler not to push so many tasks on workers? In prefect cloud I've found Task Concurrency feature, but I believe this stuff will not work with dask. Thx)
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Max Kureykin, I think it will actually work with Dask. Mapped tasks are individually submitted to Dask so each of them will be queued unless the quota frees up.
    Max Kureykin

    Max Kureykin

    1 year ago
    Yea, It has some quota inside, thats fine, I just was disappointed when I saw such tasks rain. I've faced another trouble with dask workers memory. After running the long flow workers reach their memory limits and stop. They have a lot of unmanaged memory, which I cannot clear by Python, so it is not a problem of lost/weak references object. Is there any way to clear this memory, or dump it somewhere?
    Kevin Kho

    Kevin Kho

    1 year ago
    Yeah maybe

    watch this

    video. There is an environment variable you can set