Hi there, I’m currently exploring the best way to...
# ask-community
r
Hi there, I’m currently exploring the best way to set up a hybrid work pool in Prefect and need some guidance. Specifically, I’m looking to achieve the following: 1. Hybrid Work Pool: I want to create a pool where workers can connect both locally (on on-premises servers) and on a serverless platform (Cloud Run). 2. Conditional Execution: I aim to execute jobs locally whenever possible. If local workers are unavailable, the jobs should then automatically be started on serverless infrastructure like Cloud Run. I understand that Prefect allows starting workers both locally and on cloud, but from what I’ve seen, actual execution always happens on serverless infrastructure. My goal is to prioritize local execution and only fall back to serverless when local resources are not available. Is there a configuration or best practice within Prefect that allows this type of hybrid execution model? Any advice or examples would be greatly appreciated. Thanks in advance for your help!
b
Hi Raffaele, just my two cents, but I think you could approach this like so: 1. Create two work pools, one for Cloud Run and one for Process (for job execution on-prem). Stand up a worker for each pool, one hosted on-prem, and one hosted as a Cloud Run service. 2. For each flow, create two deployments. One deployment that sends flow runs to the Cloud Run work pool (schedule set to inactive), and one that sends flow runs to the Process work pool (schedule set to active) 3. Configure a custom trigger in Prefect Cloud that fires off if more than X number of flow runs being sent to the Process pool enter a state of
Late
, as a result of Process workers becoming unavailable. The action can be to pause the schedules of the deployments going to Process pool, while simultaneously enabling the schedules of the deployments going to the Cloud Run pool
you could also alert relavent parties with a notification when this change happens, so you know what's happening
r
hi @Bianca Hoch thank you for your suggestions! I'll definitely try to go with that. As a feedback from an user, would be great if prefect, in any future release, updates the hybrid work pool to directly support this conditions. By now I'll setup everything as you have suggested ! thanks!
nod 1
🙏 1