Hello Community simple smile I have a flow that read CSV files from Google Cloud Storage and create a new table in BigQuery with this info. I created a project in Prefect Cloud and pass the credentials using a Secret variable GCP_CREDENTIALS, following the naming convention that Prefect provides so I don't have to pass this credentials manually or get them in the flow. This is working perfect. But now, I want to create another project in Prefect Cloud to have a Staging Environment and use the same flow (that I will register in Prefect Cloud with a Github Action to point the new project) to get the information from another Bucket in GCS and write in another dataset in BQ (in Google Provider will be another project, so I will need to use another Service Account Key). Is there a way to define Secrets by project in Prefect Cloud? is that is not the case, which is the best solution to approach this situation? Thanks!

Project-level secrets are not supported, unfortunately. I can totally understand you in that regard, this should be easier. There are a couple of things you could do to solve it: • if you have the budget, you may opt for an Enterprise account and get an additional tenant, which would allow you to have one for dev and another for prod • add a suffix like YOUR_SECRET_DEV to separate dev secrets in the UI - I get why this is not ideal, but might be an option This page discusses this more Also: we hear you loud and clear and this will get way easier in Prefect 2.0 prefect2

Thanks for your response @Anna Geller. So for example, having in mind that I'm working with k8s, I can create two custom images with docker using different google credentials and use them to run the flow on each environment, right?

sure, using separate images could be an option, albeit baking credentials into the image may be a security risk not yet, but we are working very actively on that abstraction, watch new Orion releases in the coming weeks and months, it's gonna be worth it 🚀

and is there a way to register a flow sending extra parameters to get them with prefect.context?

for parameters, check this post

Thanks! and can I set an env variable from the task? I want to avoid pass credentials to the read_csv method in pandas. My task is:

@task
def extract_and_load(dataset: str) -> None:
    logger = prefect.context.get("logger")
    file = f"<gs://football_transfers/transfers/{dataset}>"
    df = pd.read_csv(file)
    <http://logger.info|logger.info>("Dataset %s with %d rows loaded to DB", dataset, len(df))

for credentials, it would be better to use Secrets or some external secrets manager - but you could also use env variable if you want, you can attach those to your run configuration

Thanks!