https://prefect.io logo
Title
b

Brett Naul

02/16/2022, 4:37 PM
can post a proper issue about this but wanted to run it by folks first: I was chatting with Florian Jetter from Coiled about some performance issues and he mentioned that creating a whole lot of worker clients tends to bog down the scheduler and negatively affect stability. I noticed [this line](https://github.com/PrefectHQ/prefect/blob/master/src/prefect/executors/dask.py#L49) creates a worker client for every single task that's run, I guess to facilitate cancellation...for small flows that's probably fine, but for thousands of workers this seems to create a huge amount of extra overhead for the scheduler. I guess my q's are 1) is it absolutely necessary to create these worker clients for every task? and 2) could it be configurable somehow so that it's possible to opt-out (perhaps at the expense of less reliable task cancellation)? cc @Kevin Kho @Zanie @Chris White
z

Zanie

02/16/2022, 4:38 PM
If you can demonstrate significant improvements by disabling that code-path I’m okay with a toggle
It looks like adding a toggle would be a couple LOC
b

Brett Naul

02/16/2022, 4:46 PM
yeah it seemed like it to me too, I am running a bunch of big identical flows right now so it should be pretty easy to do a benchmark and confirm that the scheduler load goes way down; the "stability" part is a little harder to measure, the reason I looked into this is that my flows currently enter some kind of deadlocked state pretty often (75% of the time?) so I might be able to notice an anecdotal difference but not something easily reproducible
n=1 but here's disabled and enabled worker clients...75% and 100+% is the difference btwn responsive and not for my use case, so to me this seems like a great improvement
z

Zanie

02/17/2022, 5:12 PM
Sweet. I’ve opened the PR but I’ll need to find time to write a test / docs
If you want to contribute those lmk
b

Brett Naul

02/17/2022, 10:26 PM
thanks @Zanie! pretty swamped right now but I will take a look tomorrow, lmk what would be most helpful.
z

Zanie

02/18/2022, 11:53 PM
Merged now
b

Brett Naul

02/19/2022, 11:59 AM
thanks Michael!