https://prefect.io logo
Title
t

Tobias

11/11/2022, 8:25 AM
I question about working with environment variables in AWS/ECS and Python. I am working on your template. Background: I have a few environment variables that relate to Snowflake (username, password, etc). When I run the flow locally I use python dotenv to load up environment variables. This works fine when i run Prefect locally as env files are loaded from a local .env file. This wonโ€™t be the case when I deploy to ECS, so I have to define the secrets elsewhere. Aim: I want to be able to pass on secrets to the prefect environment running on ECS. Question: What is the recommended way of working with environment variables in this case? 1. When I access the AWS ECS Block I can see that adding environment variables to the Task Run is an option (see screenshot). 2. It seems possible to load credentials from a specific block (also just secrets?) using
from prefect_aws import MinIOCredentials
3. I can also see that adding Secrets to Prefect Cloud is an option in 1.0, but this seems not to be possible in 2.0?
n

Nate

11/11/2022, 4:18 PM
hey @Tobias!
Question: What is the recommended way of working with environment variables in this case?
It kind of depends in my opinion! For example, if your flow just needs access to a snowflake username / password so it can create a connection object for snowflake, I would define some SnowflakeCredentials / SnowflakeConnector blocks that you can load in to handle that for you, which you can do with
pip install prefect-snowflake

prefect block register -m prefect_snowflake.credentials
and then configuring them in the UI -- secrets do exist in prefect 2, they're just a special type of Block now. You can configure them in the UI by going to Blocks + and then load them in your flows like
from prefect.blocks.system import Secret


my_secret = Secret.load("my-secret").get()
-- however if you really just want to use env vars, I would just set as many of them as you want in the
env
field of the infrastructure block as you mentioned in part 1 of your question
๐Ÿ™Œ 1
๐Ÿ‘€ 1
t

Tobias

11/14/2022, 7:50 AM
Perfect. Thanks! I will try to make the Snowflake Block and see how that works out for me.
๐Ÿ‘ 1