Hi, everyone! I just published a multiprocess task runner for Prefect:
https://github.com/rpeden/prefect-multiprocess
I started this little side project nearly a year ago when I wanted to learn more about Prefect internals. It was primitive, but it mostly worked. I recently revisited it and found it was broken after the concurrency changes in Prefect 2.8.3. So, I decided to give it a rewrite since I had some workflows I wanted to run in parallel without needing to deal with Dask or Ray. They're great when you need them, but I wanted something a little simpler.
This task runner uses a
ProcessPoolExecutor
from the standard library, and doesn't install extra dependencies beyond what Prefect already comes with. It works quite well for my tasks, so I figured I'd share it in case it's useful for anyone else.
I've added some notes and examples in the README, so please feel free to give it a try!