My organization today is using a home-made data orchestration tool built on top of RQ in Python. I'm evaluating Prefect as an alternative, but from what I can tell workers are blocking. Is my understanding correct?
E.g. a flow should run 10 tasks, some of which wait for the results of others. If I only have a single worker, is it impossible for me to be running 2 instances of this flow concurrently?
✅ 1
j
Jake Kaplan
09/16/2023, 2:58 PM
Hi, you can absolutely run two instances of a flow at once if you want to.
You can sort of think of Workers as flow deployers. They don’t run the flow themselves, they look for scheduled work and then deploy flows to a processs, k8s, ecs, etc. (whatever worker type you choose) and keep an eye on it.
So for example if you are using a ProcessWorker, it would run two runs of your flow each in their own process. Hope that helps!
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.