Hi, everyone! I just published a multiprocess task...
# show-and-tell
r
Hi, everyone! I just published a multiprocess task runner for Prefect: https://github.com/rpeden/prefect-multiprocess I started this little side project nearly a year ago when I wanted to learn more about Prefect internals. It was primitive, but it mostly worked. I recently revisited it and found it was broken after the concurrency changes in Prefect 2.8.3. So, I decided to give it a rewrite since I had some workflows I wanted to run in parallel without needing to deal with Dask or Ray. They're great when you need them, but I wanted something a little simpler. This task runner uses a
ProcessPoolExecutor
from the standard library, and doesn't install extra dependencies beyond what Prefect already comes with. It works quite well for my tasks, so I figured I'd share it in case it's useful for anyone else. I've added some notes and examples in the README, so please feel free to give it a try!
🙌 14
🚀 5
o
Looks useful, will check it out! Thanks.
c
This is great 👍 was thinking about this the other day actually
Also your prefect docker compose is perfect 👍