https://prefect.io logo
g

Gregory Hunt

09/18/2023, 7:43 PM
Prefect Devs, I am trying to writing a custom process to push flow_runs to a group of worker via pub/sub style approach. I currently have a group of ProcessWorkers, but they trip over each other because they are polling from the same queue. I am looking for a way to push those flows to the ProcessWorkers, instead of them polling
@alex @Dylan any ideas
d

Dylan

09/19/2023, 3:23 PM
Hey Gregory, You can check out Push Work Pools for ways to push flow runs to serverless infrastructure. You could also define Work Queues with different priorities to try to help your process workers parse out work a bit better. I don't think we support a push pattern with process workers exactly in the way you've outlined at the moment, but you could set up a per-flow-queue using flow.serve instead of process workers. As a small note, please don't ping specific developers in the community support channel. We'll do our best to help everyone in our community when we can. If you need dedicated support offerings with SLAs, please reach out to us at sales@prefect.io to explore options. Happy Engineering!