https://prefect.io logo
#prefect-community
Title
# prefect-community
t

Tim Wright

04/29/2022, 1:43 PM
I have a question related to Prefect Cloud. We have our jobs running in AKS - with our Flows using a custom Image. Our flows are all built into the image using Docker Storage. We've been trying to mitigate the possibility of adverse effects if we were somehow to leak or Prefect API keys. To that end we've locked down our cluster a bit with a security policy to disallow any flows trying to run on images that we have not authored (e.g. in our registry). Is there any possibility that someone (with Prefect Cloud API Keys) could register a flow that uses our image, but an alternate flow storage mechanism? We were considering subclassing the FlowRunner class to handle some checking/validation there - validating some of the metadata set at registration time against some of the metadata provided at Runtime. Is this possible, does it make sense? Is there a way we can ensure that our Custom FlowRunner is the default (and only) FlowRunner permitted? To be more concise our goal is to ensure that the only flows running in our infra (AKS) run with our images (this we have solved), and only the flows that are built into those images can be run (this we have not solved).
h

Henning Holgersen

04/29/2022, 2:17 PM
Very good question, I have thought about this too. The API tokens are short and difficult to rotate, and they are all that's needed for remote code execution. One possible solution is to lock down your agent host with firewall/network rules, so that it does not have access to images or code in any location that's not controlled by you. This requires keeping a copy of the prefect image in your own repo, but that should be fairly manageable. In my case I use Docker for storage, so it's a single repo domain that needs to be allowed.
t

Tim Wright

04/29/2022, 3:42 PM
Is this something you have done @Henning Holgersen? Do you just allow the Agent to communicate out to Prefect Cloud and your own registry? Then your flow execution environment has access to all the services you need to orchestrate?
a

Anna Geller

04/29/2022, 3:52 PM
only the flows that are built into those images can be run (this we have not solved)
Have you thought about incorporating that as some sort of integration test in your CI? Not sure I fully understood your use case - is your question purely about permissions with respect to who (or which process) is allowed to register flows? or who is allowed to push images to your registry?
h

Henning Holgersen

04/29/2022, 3:54 PM
Currently I have also allowed docker hub and GitHub because we are only in a dev environment, meaning that we are more relaxed than what I mentioned. But basically yes, we have explicitly opened the stuff we need, everything else is blocked even outbound. Some DBs, specific azure storage and a few other trusted endpoints are allowed. So far it works nicely.
🙌 2
t

Tim Wright

04/29/2022, 4:00 PM
@Anna Geller Neither. The concern is that the only thing controlling who can register a flow with Prefect cloud is our API Keys. If an API key becomes compromised someone could potentially register a flow to Prefect cloud (outside of our CI). Inside our AKS cluster we ensure via policy that only our image can be used for flows - so not worried about that piece. I was concerned about the possibility that a nefarious actor (with access to valid Prefect Cloud keys) could register a flow that uses our image for execution, but uses an alternate flow storage mechanism. The thought is that our Agent might pick up the flow to run, use our image (which is permitted by policy) but use some alternate flow storage mechanism that could permit a flow may be pulled into our image at runtime. Not sure if that makes sense or not
a

Anna Geller

04/29/2022, 5:28 PM
But this only means that you need to keep this secret secure the same way as any API key. Using API keys and tokens is quite a standard practice; a similar approach is used e.g. with GitHub Personal Access Tokens or docker container registry password. Are you on the Enterprise Cloud tier? If so, you could contact sales@prefect.io and there are various ways you can grant various RBAC permissions to different user roles to further lock down access.
t

Tim Wright

04/29/2022, 5:51 PM
@Anna Geller completely understood, and we don't plan to lose those credentials. We are trying to understand/mitigate the impacts if it did happen.
👍 1
12 Views