Hello! I was wondering if it's possible to set up a Prefect Cloud Workpool with workers from separate Kubernetes instances, or otherwise, to split up work between two Kubernetes locations.
For my use case, I want to have redundancy in case a physical Kubernetes cluster fails. My desire on how to accomplish that is to have workers in two locations/clusters which can both pick up a deployment's work. If one of the locations were to go down, the other would seamlessly continue working fine.
Is this something that's possible? My thought would be to configure a work pool such that it has workers from multiple locations, I'm just not sure if that's actually supported (and if so, how to do it).
✅ 1
j
Jake Kaplan
01/30/2024, 3:42 PM
Hey that is supported. You would create one work pool and then run two workers both pointed at that same work pool.
Both workers are able to pickup the job but you'll see that only one worker is allowed to claim the job
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.