Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi all. Does Prefect support isolated workers? Meaning, a single master for all pipelines, but then separate physical workers (EC2 instances that are in different securitygroups, etc.) that only have access to the databases, etc. that they need access to? Part of our business requirements is single tenant for data pipelines.

Hi <@U010DBJRVQD>, I believe you can accomplish this task with some smart permissions, multiple agents, and flow affinity. Essentially:
1. Prefect Cloud can orchestrate all flows in all security groups
2. For each security-group / isolated place you want to run flows, create an agent with a particular label
3. Flows registered with that label can only be run by agents with that label
<https://docs.prefect.io/cloud/agents/overview.html#flow-affinity-labels>

Let me know if I can provide more detail :smile:

So the Preflect Cloud ("master"?) wouldn't pass a job to an agent unless it had that label?

This is what I do for our data warehousing work internally (I’m actually working on a blog post about this)

Thanks! I think that blog post could be valuable!

How to replicate Jenkins "labels" in Prefect

Haha the post is about our whole data warehousing setup using Prefect but this is definitely an important part! Maybe I’ll add more detail about this than I was originally planning