Ryan Peden

08/22/2023, 2:14 AM
Hi, everyone! I just published a multiprocess task runner for Prefect: I started this little side project nearly a year ago when I wanted to learn more about Prefect internals. It was primitive, but it mostly worked. I recently revisited it and found it was broken after the concurrency changes in Prefect 2.8.3. So, I decided to give it a rewrite since I had some workflows I wanted to run in parallel without needing to deal with Dask or Ray. They're great when you need them, but I wanted something a little simpler. This task runner uses a
from the standard library, and doesn't install extra dependencies beyond what Prefect already comes with. It works quite well for my tasks, so I figured I'd share it in case it's useful for anyone else. I've added some notes and examples in the README, so please feel free to give it a try!
🚀 2
🙌 10

Oscar Björhn

08/22/2023, 10:52 AM
Looks useful, will check it out! Thanks.

Cody Webb

08/22/2023, 10:27 PM
This is great 👍 was thinking about this the other day actually
Also your prefect docker compose is perfect 👍