Hi all Does Prefect support isolated workers Meaning a singl

Question

Hi all Does Prefect support isolated workers Meaning a single master for all pipelines but then separate physical workers EC2 instances that are in different securitygroups etc that only have access to the databases etc that they need access to Part of our business requirements is single tenant for data pipelines

Dylan · Answer

Hi < Matt Juszczak> I believe you can accomplish this task with some smart permissions multiple agents and flow affinity Essentially 1 Prefect Cloud can orchestrate all flows in all security groups 2 For each security group isolated place you want to run flows create an agent with a particular label 3 Flows registered with that label can only be run by agents with that label

Dylan · Answer

Let me know if I can provide more detail smile

Matt Juszczak · Answer

Thanks

Matt Juszczak · Answer

So the Preflect Cloud master wouldn t pass a job to an agent unless it had that label

Dylan · Answer

Correct

Dylan · Answer

This is what I do for our data warehousing work internally I m actually working on a blog post about this

Matt Juszczak · Answer

Thanks I think that blog post could be valuable

Matt Juszczak · Answer

How to replicate Jenkins labels in Prefect

Dylan · Answer

Haha the post is about our whole data warehousing setup using Prefect but this is definitely an important part Maybe I ll add more detail about this than I was originally planning

Matt Juszczak · Answer

Thanks Sounds good and makes sense