b

    Brett Naul

    7 months ago
    can post a proper issue about this but wanted to run it by folks first: I was chatting with Florian Jetter from Coiled about some performance issues and he mentioned that creating a whole lot of worker clients tends to bog down the scheduler and negatively affect stability. I noticed [this line](https://github.com/PrefectHQ/prefect/blob/master/src/prefect/executors/dask.py#L49) creates a worker client for every single task that's run, I guess to facilitate cancellation...for small flows that's probably fine, but for thousands of workers this seems to create a huge amount of extra overhead for the scheduler. I guess my q's are 1) is it absolutely necessary to create these worker clients for every task? and 2) could it be configurable somehow so that it's possible to opt-out (perhaps at the expense of less reliable task cancellation)? cc @Kevin Kho @Michael Adkins @Chris White
    Michael Adkins

    Michael Adkins

    7 months ago
    If you can demonstrate significant improvements by disabling that code-path I’m okay with a toggle
    It looks like adding a toggle would be a couple LOC
    b

    Brett Naul

    7 months ago
    yeah it seemed like it to me too, I am running a bunch of big identical flows right now so it should be pretty easy to do a benchmark and confirm that the scheduler load goes way down; the "stability" part is a little harder to measure, the reason I looked into this is that my flows currently enter some kind of deadlocked state pretty often (75% of the time?) so I might be able to notice an anecdotal difference but not something easily reproducible
    n=1 but here's disabled and enabled worker clients...75% and 100+% is the difference btwn responsive and not for my use case, so to me this seems like a great improvement
    Michael Adkins

    Michael Adkins

    7 months ago
    Sweet. I’ve opened the PR but I’ll need to find time to write a test / docs
    If you want to contribute those lmk
    b

    Brett Naul

    7 months ago
    thanks @Michael Adkins! pretty swamped right now but I will take a look tomorrow, lmk what would be most helpful.
    Michael Adkins

    Michael Adkins

    7 months ago
    Merged now
    b

    Brett Naul

    7 months ago
    thanks Michael!