Hi team,
I'm new to prefect (loving it so far) bu...
# ask-communityf
FB
12/11/2023, 12:58 AMHi team,
I'm new to prefect (loving it so far) but have some questions about the security model. For some background context: we provide services in a security-restricted industry where we provide services to large enterprises. Our services are deployed via Kubernetes, using k8s namespaces to provide (execution and network) isolation between our customers - ie: if a container somehow got breached for one customer, it wouldn't allow that container to reach into another customers network, database, etc.
We wish to run Prefect to orchestrate jobs, but have run into a problem where it appears it's impossible to do so without exposing ourselves to significant security risk (if I understand correctly).
See continuation of thread attached to this post for more details (so it does't get too long in channel).
FB
12/11/2023, 12:59 AMThe issue is as follows - we have two options:
1. Use OSS prefect server;
2. Use prefect cloud;
Our initial approach was to evaluate OSS server, however, the lack of RBAC presents challenges:
FB
12/11/2023, 1:00 AMSpecifically, no matter how you architect it, the flows executing always have the ability to talk back to the server API without restriction.
The architecture I trialled was isolating the server and worker in it's own namespace and then giving it's service account permissions to launch kubernetes jobs in the per-customer k8s namespaces, such that the flows/jobs have access to their appropriate resources (databases/etc).
FB
12/11/2023, 1:02 AMHowever, because the prefect jobs require access to the API (to presumably push logs/etc back up to the server & UI), it also means they have the ability to register/execute their own flows (as there's no RBAC), which is effectively a Remote Code Execution (or at least sandbox escape) vulnerability against remote namespaces.
To be more clear: Any container in any namespace could schedule any code to execute in a foreign namespace via the Prefect server API.
FB
12/11/2023, 1:04 AMThe other option is to switch to Prefect Cloud (though it's slightly unclear from the docs whether RBAC would solve this), however, this is not a feasible option for us, as we would require all of our customers to sign-off on exposing our (and their) entire infrastructure to Prefect's security, for much the same reason, ie:
If Prefect were to be breached, the cloud API allows full, unrestricted code execution in the worker environment (which is effectively as privileged as breaching the full app infrastructure).
FB
12/11/2023, 1:05 AMAm I interpreting this correctly? Am I correct in thinking that Prefect is not really suitable for a high-security multi-tenant enterprise environment or am I missing something?
I really don't want to switch to airflow (though it supports this security model).
Very much appreciate any help!
m
Miranda Byrd
12/11/2023, 4:32 AMTagging to follow (and my sympathies for potential airflow consequences)
j
juan suarez (hf)
12/11/2023, 2:23 PMHey @FB
We have the same issues and we decided to run with option of a multi-account / multi-tenant solution where each customer has their own orchestrator
juan suarez (hf)
12/11/2023, 2:25 PMWe have then created a custom Docker file with security (API / login) before reaching the Orchestrator. The problem is that once that the person has reach the orchestrator, they have access to all the pipelines, so there’s no actual solution for that.
A not-so-complex CI/CD process allows us to update the Prefect version for our customers while keeping the isolation and security above mentioned
f
FB
12/12/2023, 12:09 AMThanks for the response @juan suarez (hf) - I'm glad it's not just me! I also considered "orchestrator per customer" but it seems like a bit of an unfortunate hack.
FB
12/12/2023, 12:09 AMAlso, like you said, you still have access to everything once you get to the orchestrator.
FB
12/12/2023, 12:18 AMRather than extending the server app source, I'm considering building a simple/custom ingress controller that would add basic RBAC to the server API (essentially an API Gateway) - basically, you restrict workers to only be able to access the worker API methods, exactly the same as with Prefect cloud.
You'd then just need to issue workers with an appropriate API key that only has access to the appropriate methods.
You could also lock down the web UI using the same basic pattern (just block all POSTs).
Happy to opensource my efforts, though I'm keenly aware this is just rebuilding part of Prefect's paid offering, which I don't want to do (I just don't really have an option other than moving to airflow).
Would love if someone from prefect could possibly chime in on this one? @Will Raphaelson?
w
Will Raphaelson
12/12/2023, 12:39 AMThanks FB. You’re correct that youd essentially be rebuilding part of prefect clouds backend, but I understand your constraints. Let me circle up with some folks internally and get an answer back to you in the next few days.
f
FB
12/12/2023, 2:14 AMThanks Will, really appreciate it!
j
juan suarez (hf)
12/12/2023, 8:46 AMWe excluded that possibility @FB because you don’t have true isolation if you do that. And you can effectively find yourself with shared blocks or shared elements that could compromise the data isolation. But I’ll be happy to see what solution you build
f
FB
12/12/2023, 8:13 PMYeah, it's a valid point, @juan suarez (hf). I guess we could always go one-orchestrator-per-namespace like you've ended up with but it seems kinda messy/duplicative.
j
juan suarez (hf)
12/12/2023, 8:14 PMIt’s messy, but it’s the best solution that we came up with.
f
FB
12/18/2023, 4:57 AMHi @Will Raphaelson, any updates on this one perchance?
w
Will Raphaelson
12/18/2023, 3:57 PMHey FB, sorry for the delay. Typically, users with similar security constraints would use Prefect Cloud, the RBAC functionality of which would meet your criteria. Customers typically inform their users of this via a change to their subprocessor list, like we do here.
If this is a nonstarter, building your own RBAC layer seems like a good direction to go explore, though I cant much consult on that as its outside my domain. We also, in special cases, chat with folks about a cloud deployment on your own infrastructure. If this is of interest, would you DM me and I can connect you to the right folks?