Hi, everyone! I just published a multiprocess task runner for Prefect: https://github.com/rpeden/prefect-multiprocess
I started this little side project nearly a year ago when I wanted to learn more about Prefect internals. It was primitive, but it mostly worked. I recently revisited it and found it was broken after the concurrency changes in Prefect 2.8.3. So, I decided to give it a rewrite since I had some workflows I wanted to run in parallel without needing to deal with Dask or Ray. They're great when you need them, but I wanted something a little simpler.
This task runner uses a
ProcessPoolExecutor
from the standard library, and doesn't install extra dependencies beyond what Prefect already comes with. It works quite well for my tasks, so I figured I'd share it in case it's useful for anyone else.
I've added some notes and examples in the README, so please feel free to give it a try!
🙌 14
🚀 5
o
Oscar Björhn
08/22/2023, 10:52 AM
Looks useful, will check it out! Thanks.
c
Cody Webb
08/22/2023, 10:27 PM
This is great 👍 was thinking about this the other day actually
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.