https://prefect.io logo
m

Matt Juszczak

03/23/2020, 4:32 PM
Hi all. Does Prefect support isolated workers? Meaning, a single master for all pipelines, but then separate physical workers (EC2 instances that are in different securitygroups, etc.) that only have access to the databases, etc. that they need access to? Part of our business requirements is single tenant for data pipelines.
d

Dylan

03/23/2020, 5:24 PM
Hi @Matt Juszczak, I believe you can accomplish this task with some smart permissions, multiple agents, and flow affinity. Essentially: 1. Prefect Cloud can orchestrate all flows in all security groups 2. For each security-group / isolated place you want to run flows, create an agent with a particular label 3. Flows registered with that label can only be run by agents with that label https://docs.prefect.io/cloud/agents/overview.html#flow-affinity-labels
upvote 1
Let me know if I can provide more detail 😄
m

Matt Juszczak

03/23/2020, 5:54 PM
Thanks!
So the Preflect Cloud ("master"?) wouldn't pass a job to an agent unless it had that label?
d

Dylan

03/23/2020, 5:55 PM
Correct
This is what I do for our data warehousing work internally (I’m actually working on a blog post about this)
m

Matt Juszczak

03/23/2020, 5:57 PM
Thanks! I think that blog post could be valuable!
marvin 2
How to replicate Jenkins "labels" in Prefect
d

Dylan

03/23/2020, 5:59 PM
Haha the post is about our whole data warehousing setup using Prefect but this is definitely an important part! Maybe I’ll add more detail about this than I was originally planning
m

Matt Juszczak

03/23/2020, 6:21 PM
Thanks! Sounds good, and makes sense!