Hi guys! The question is about prefect/dask combo. Is there any way to limit dask scheduler not to push so many tasks on workers? In prefect cloud I've found Task Concurrency feature, but I believe this stuff will not work with dask. Thx)

Hey @Max Kureykin, I think it will actually work with Dask. Mapped tasks are individually submitted to Dask so each of them will be queued unless the quota frees up.

Yea, It has some quota inside, thats fine, I just was disappointed when I saw such tasks rain. I've faced another trouble with dask workers memory. After running the long flow workers reach their memory limits and stop. They have a lot of unmanaged memory, which I cannot clear by Python, so it is not a problem of lost/weak references object. Is there any way to clear this memory, or dump it somewhere?

Yeah maybe video. There is an environment variable you can set