https://prefect.io logo
Title
m

Matt Camp

04/28/2021, 12:58 PM
hello, I'm new to Prefect, just got it running yesterday. I am coming from a history of using Airflow and was curious how Prefect users manage variable and connetions? I saw on the main Prefect docs that secrets aren't supported by the
prefect-server
and that you need to use the
config.yml
. This seems like an all around disaster waiting to happen. I am not able to use the Prefect Cloud due to where I am setting things up, so are there alternative methods people use? TIA
🙏 1
k

Kevin Kho

04/28/2021, 1:27 PM
Hey @Matt Camp! When it comes to connections, what Prefect users do is load them from a file that has all of them. You can also use environment variables. If you need something more secure, you’d probably have to use a Vault type of service. Will double check.
Where are you running your flows? You could set the env variables there.
m

Matt Camp

04/28/2021, 1:30 PM
currently its on a single node, but we intend to scale to k8s once we figure out how everything works. I saw the Vault github but didn't poke too hard on that. Is that something we could self host and if so is there some config in Prefect that we use to point it toward our local Vault?
pretty much my two big concerns with using mutliple
config.yml
or environment variables is there isn't a great way I know of to sync them between workers/agents. Say I have DB1 and it has a user/pass and I have DB2 with another user/pass. Then I have a bunch of different flows all using those DBs. What if we had to change the password on one, or we added another DB. We would with my current mental model of how things work, have to go to every Agent/Flow and update the config as apposed to a single central place. At scale this seems like a bad idea. But I am still learning a lot of this stuff so maybe its not as bad in practice?
k

Kevin Kho

04/28/2021, 1:35 PM
Yes Vault has a self-manged open source tier. I don’t believe there is a built-in config with server. But I don’t think that would stop you from connecting once the Vault is set up.
I think what some users do is have a central CI/CD process be responsible for pushing to the Agents and different environments. I think in this case, it might be possible to have the secret on the CI/CD tool. Of course, this is highly specific to your use case
🙌 1
m

Matt Camp

04/28/2021, 1:38 PM
thank you! I'll give all that a try!
👍 1
z

Zanie

04/28/2021, 2:18 PM
We're also interested in adding a generic Vault task that allows users to pull secrets from vault in their own infrastructure. However, you'll still need to provide the Vault authentication secrets somehow (ie via env variable)
d

davzucky

04/29/2021, 2:47 AM
We are using in kuberbetes sealed secret and external secret operator to get our secret. I wrote task kubernetessecret that you can use to retrieve them from your flow https://docs.prefect.io/api/latest/tasks/kubernetes.html#replacenamespacedservice Hope that help
👍 2
r

Rob Fowler

05/01/2021, 9:26 AM
I am using vault for everything here at MCS. I use the k8s vault service account authentication. (For non prefect long running processes, but easily done as well, we have a vault sidecar in the deployment). At first I wrote a VaultGetSecret() task and used that to get secrets but it seemed a waste so now I have a class that all tasks that need vault secrets derive from. ie. class VaultTask(Task): ..
using vault in prefect is trivial but authenticating it not as simple. I banned injecting tokens as environment variables. Vault tokens are too short lived, for good reason.
a

Aurélien Vallée

05/12/2021, 5:24 AM
FWIW you can use AWS SSM to pull credentials if you're using AWS