can post a proper issue about this but wanted to run it by folks first: I was chatting with Florian Jetter from Coiled about some performance issues and he mentioned that creating a whole lot of worker clients tends to bog down the scheduler and negatively affect stability. I noticed [this line](https://github.com/PrefectHQ/prefect/blob/master/src/prefect/executors/dask.py#L49) creates a worker client for every single task that's run, I guess to facilitate cancellation...for small flows that's probably fine, but for thousands of workers this seems to create a huge amount of extra overhead for the scheduler. I guess my q's are 1) is it absolutely necessary to create these worker clients for every task? and 2) could it be configurable somehow so that it's possible to opt-out (perhaps at the expense of less reliable task cancellation)? cc @Kevin Kho @Zanie @Chris White

If you can demonstrate significant improvements by disabling that code-path I’m okay with a toggle

yeah it seemed like it to me too, I am running a bunch of big identical flows right now so it should be pretty easy to do a benchmark and confirm that the scheduler load goes way down; the "stability" part is a little harder to measure, the reason I looked into this is that my flows currently enter some kind of deadlocked state pretty often (75% of the time?) so I might be able to notice an anecdotal difference but not something easily reproducible

Sweet. I’ve opened the PR but I’ll need to find time to write a test / docs

thanks @Zanie! pretty swamped right now but I will take a look tomorrow, lmk what would be most helpful.

Merged now

thanks Michael!