https://prefect.io logo
i

Ievgenii Martynenko

02/03/2022, 2:26 PM
Hi, Maybe someone can help on how to do best options in such case 1) We have a linux docker image as Prefect Agent with mounted S3 storage acting as primary storage for Flows, configs and some resources used for tasks. 2) Each Flow is executed in separate container using DockerRun(image=prefecthq/prefect:0.15.13-python3.8, host_config = host_config). So we have isolated Flow Containers inside Agent Container :). host_config is used to mount Agents S3 into Flow runtime, so that Flow, when running can pick up configs and resources mounted to Agent. Tricky thing is how to pass connection strings inside Flow Container? 1) I thought that we can pass them as arguments of DockerRun(.... environment = environment) according to (https://docker-py.readthedocs.io/en/stable/api.html#module-docker.api.container), but we need to prepare the list and it means that it will be "plain text". Not good. 2) Another option is to define some dictionary in the Flow and replace variables in CI/CD tool during deployment, but in this case it will be stored as plain text on storage inside Flow. This might be an option in case if serialized Prefect flow can be encrypted. 3) ??? maybe some other ideas?
3) Option is to define Env variables (as well as pip dependencies) in docker image used in DockerRun() and that makes a trick, but in case if any Env variable changes, we need to re-build/ republish whole docker image.
k

Kevin Kho

02/03/2022, 3:02 PM
Are you on Server or Cloud? Cuz for Cloud I think the answer here is Prefect Secrets?
i

Ievgenii Martynenko

02/03/2022, 3:05 PM
On Server. For Cloud, yes, Secrets is an option (if your SecOps allows this).
k

Kevin Kho

02/03/2022, 3:09 PM
Ok I think best practice you be using some kind of service similar to Prefect Secrets. I have seen someone use AWS Secrets Manager. Just turn off
checkpointing
for tasks that retrieve secrets so they are not persisted. Your other option is adding it to the agent
prefect … agent start --env ..
but I don’t think you’ll like that either.
i

Ievgenii Martynenko

02/03/2022, 3:12 PM
In other words, extend Secrets functionality to query external Vault and pass them into Flow/Task directly on runtime.
k

Kevin Kho

02/03/2022, 3:14 PM
I personally think so. Other people use KubernetesSecrets, just some kind of real secret handler.
3 Views