hello I m new to Prefect just got it running yesterday I am

Question

hello I m new to Prefect just got it running yesterday I am coming from a history of using Airflow and was curious how Prefect users manage variable and connetions I saw on the main Prefect docs that secrets aren t supported by the `prefect server` and that you need to use the `config yml` This seems like an all around disaster waiting to happen I am not able to use the Prefect Cloud due to where I am setting things up so are there alternative methods people use TIA

Kevin Kho · Answer

Hey < Matt Camp> When it comes to connections what Prefect users do is load them from a file that has all of them You can also use environment variables If you need something more secure you d probably have to use a Vault type of service Will double check

Kevin Kho · Answer

Where are you running your flows You could set the env variables there

Matt Camp · Answer

currently its on a single node but we intend to scale to k8s once we figure out how everything works I saw the Vault github but didn t poke too hard on that Is that something we could self host and if so is there some config in Prefect that we use to point it toward our local Vault

Matt Camp · Answer

pretty much my two big concerns with using mutliple `config yml` or environment variables is there isn t a great way I know of to sync them between workers agents Say I have DB1 and it has a user pass and I have DB2 with another user pass Then I have a bunch of different flows all using those DBs What if we had to change the password on one or we added another DB We would with my current mental model of how things work have to go to every Agent Flow and update the config as apposed to a single central place At scale this seems like a bad idea But I am still learning a lot of this stuff so maybe its not as bad in practice

Kevin Kho · Answer

Yes Vault has a <https www hashicorp com products vault pricing|self manged> open source tier I don t believe there is a built in config with server But I don t think that would stop you from connecting once the Vault is set up

Kevin Kho · Answer

I think what some users do is have a central CI CD process be responsible for pushing to the Agents and different environments I think in this case it might be possible to have the secret on the CI CD tool Of course this is highly specific to your use case

Matt Camp · Answer

thank you I ll give all that a try

Zanie · Answer

We re also interested in adding a generic Vault task that allows users to pull secrets from vault in their own infrastructure However you ll still need to provide the Vault authentication secrets somehow ie via env variable

davzucky · Answer

We are using in kuberbetes sealed secret and external secret operator to get our secret I wrote task kubernetessecret that you can use to retrieve them from your flow <https docs prefect io api latest tasks kubernetes html replacenamespacedservice|https docs prefect io api latest tasks kubernetes html replacenamespacedservice> Hope that help

Rob Fowler · Answer

I am using vault for everything here at MCS I use the k8s vault service account authentication For non prefect long running processes but easily done as well we have a vault sidecar in the deployment At first I wrote a VaultGetSecret task and used that to get secrets but it seemed a waste so now I have a class that all tasks that need vault secrets derive from ie class VaultTask Task

Rob Fowler · Answer

using vault in prefect is trivial but authenticating it not as simple I banned injecting tokens as environment variables Vault tokens are too short lived for good reason

Aurélien Vallée · Answer

FWIW you can use AWS SSM to pull credentials if you re using AWS