https://prefect.io logo
Title
m

Mateo Merlo

05/11/2022, 4:31 PM
Hello Community 😒imple_smile: I have a flow that read CSV files from Google Cloud Storage and create a new table in BigQuery with this info. I created a project in Prefect Cloud and pass the credentials using a Secret variable GCP_CREDENTIALS, following the naming convention that Prefect provides so I don't have to pass this credentials manually or get them in the flow. This is working perfect. But now, I want to create another project in Prefect Cloud to have a Staging Environment and use the same flow (that I will register in Prefect Cloud with a Github Action to point the new project) to get the information from another Bucket in GCS and write in another dataset in BQ (in Google Provider will be another project, so I will need to use another Service Account Key). Is there a way to define Secrets by project in Prefect Cloud? is that is not the case, which is the best solution to approach this situation? Thanks!
a

Anna Geller

05/11/2022, 5:09 PM
Project-level secrets are not supported, unfortunately. I can totally understand you in that regard, this should be easier. There are a couple of things you could do to solve it: • if you have the budget, you may opt for an Enterprise account and get an additional tenant, which would allow you to have one for dev and another for prod • add a suffix like YOUR_SECRET_DEV to separate dev secrets in the UI - I get why this is not ideal, but might be an option This page discusses this more Also: we hear you loud and clear and this will get way easier in Prefect 2.0 😛refect2:
m

Mateo Merlo

05/11/2022, 5:35 PM
Thanks for your response @Anna Geller. So for example, having in mind that I'm working with k8s, I can create two custom images with docker using different google credentials and use them to run the flow on each environment, right?
Btw, this is an option that is already available in Prefect 2.0? We are just starting with our ELT and Prefect integration so perhaps is better start to use directly Prefect 2.0
a

Anna Geller

05/11/2022, 5:40 PM
sure, using separate images could be an option, albeit baking credentials into the image may be a security risk not yet, but we are working very actively on that abstraction, watch new Orion releases in the coming weeks and months, it's gonna be worth it 🚀
🙌 1
m

Mateo Merlo

05/11/2022, 5:47 PM
and is there a way to register a flow sending extra parameters to get them with prefect.context?
I will follow your suggestion adding a suffix to VARIABLES, but I need a way to differentiate which is the current project that is running the flow, to be able to choose the correct var secret
👍 1
a

Anna Geller

05/11/2022, 7:22 PM
for parameters, check this post
m

Mateo Merlo

05/11/2022, 8:07 PM
Thanks! and can I set an env variable from the task? I want to avoid pass credentials to the read_csv method in pandas. My task is:
@task
def extract_and_load(dataset: str) -> None:
    logger = prefect.context.get("logger")
    file = f"<gs://football_transfers/transfers/{dataset}>"
    df = pd.read_csv(file)
    <http://logger.info|logger.info>("Dataset %s with %d rows loaded to DB", dataset, len(df))
a

Anna Geller

05/12/2022, 1:06 AM
for credentials, it would be better to use Secrets or some external secrets manager - but you could also use env variable if you want, you can attach those to your run configuration
m

Mateo Merlo

05/12/2022, 7:37 AM
Thanks!