https://prefect.io logo
Join Slack
Powered by
hello, I'm new to Prefect, just got it running yes...
# prefect-server
m
hello, I'm new to Prefect, just got it running yesterday. I am coming from a history of using Airflow and was curious how Prefect users manage variable and connetions? I saw on the main Prefect docs that secrets aren't supported by the 
prefect-server
and that you need to use the 
config.yml
. This seems like an all around disaster waiting to happen. I am not able to use the Prefect Cloud due to where I am setting things up, so are there alternative methods people use? TIA
🙏 1
k
Hey @Matt Camp! When it comes to connections, what Prefect users do is load them from a file that has all of them. You can also use environment variables. If you need something more secure, you’d probably have to use a Vault type of service. Will double check.
Where are you running your flows? You could set the env variables there.
m
currently its on a single node, but we intend to scale to k8s once we figure out how everything works. I saw the Vault github but didn't poke too hard on that. Is that something we could self host and if so is there some config in Prefect that we use to point it toward our local Vault?
pretty much my two big concerns with using mutliple 
config.yml
or environment variables is there isn't a great way I know of to sync them between workers/agents. Say I have DB1 and it has a user/pass and I have DB2 with another user/pass. Then I have a bunch of different flows all using those DBs. What if we had to change the password on one, or we added another DB. We would with my current mental model of how things work, have to go to every Agent/Flow and update the config as apposed to a single central place. At scale this seems like a bad idea. But I am still learning a lot of this stuff so maybe its not as bad in practice?
k
Yes Vault has a self-manged open source tier. I don’t believe there is a built-in config with server. But I don’t think that would stop you from connecting once the Vault is set up.
I think what some users do is have a central CI/CD process be responsible for pushing to the Agents and different environments. I think in this case, it might be possible to have the secret on the CI/CD tool. Of course, this is highly specific to your use case
🙌 1
m
thank you! I'll give all that a try!
đź‘Ť 1
z
We're also interested in adding a generic Vault task that allows users to pull secrets from vault in their own infrastructure. However, you'll still need to provide the Vault authentication secrets somehow (ie via env variable)
d
We are using in kuberbetes sealed secret and external secret operator to get our secret. I wrote task kubernetessecret that you can use to retrieve them from your flow https://docs.prefect.io/api/latest/tasks/kubernetes.html#replacenamespacedservice Hope that help
đź‘Ť 2
r
I am using vault for everything here at MCS. I use the k8s vault service account authentication. (For non prefect long running processes, but easily done as well, we have a vault sidecar in the deployment). At first I wrote a VaultGetSecret() task and used that to get secrets but it seemed a waste so now I have a class that all tasks that need vault secrets derive from. ie. class VaultTask(Task): ..
using vault in prefect is trivial but authenticating it not as simple. I banned injecting tokens as environment variables. Vault tokens are too short lived, for good reason.
a
FWIW you can use AWS SSM to pull credentials if you're using AWS
3 Views
Open in Slack
PreviousNext
Powered by