Hi there, Trying to save a csv to GCS using pandas in a Prefect cloud flow... I have set the GCP_CREDENTIALS key on prefect cloud, which I am beginning to realize is for prefect GCP tasks only, but I am getting the GCS 401 error that suggests the credentials are missing. Is there a way to pass the credentials into the pandas operation using the saved credentials on Prefect Cloud or is there another way using Prefect tasks that could work?
k
Kevin Kho
02/03/2022, 3:13 PM
You are right
GCP_CREDENTIALS
is Prefect specific. The standard way is having an env variable called
GOOGLE_APPLICATION_CREDENTIALS
and pointing it to a JSON containing the credentials.
Or maybe you can just import this function and create the Client the same way for the task you have so that you can reuse the same Secret?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.