My organization today is using a home-made data orchestration tool built on top of RQ in Python. I'm evaluating Prefect as an alternative, but from what I can tell workers are blocking. Is my understanding correct?
E.g. a flow should run 10 tasks, some of which wait for the results of others. If I only have a single worker, is it impossible for me to be running 2 instances of this flow concurrently?
09/16/2023, 2:58 PM
Hi, you can absolutely run two instances of a flow at once if you want to.
You can sort of think of Workers as flow deployers. They don’t run the flow themselves, they look for scheduled work and then deploy flows to a processs, k8s, ecs, etc. (whatever worker type you choose) and keep an eye on it.
So for example if you are using a ProcessWorker, it would run two runs of your flow each in their own process. Hope that helps!