Loic M
01/15/2021, 5:36 PMgoogle.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see <https://cloud.google.com/docs/authentication/getting-started|https://cloud.google.com/docs/authentication/getting->started
Is there something I am missing there ? Should I include the google credentials when building my flow's docker image ?Josh
01/15/2021, 5:39 PMnicholas
Loic M
01/16/2021, 1:32 PMGCSResult
My code looks like this
import prefect
from prefect.engine.results import GCSResult
from prefect.run_configs import DockerRun
from prefect.storage import Docker
@prefect.task(checkpoint=True, result =GCSResult(bucket="redacted"))
def task1():
...
storage = DockerStorage(...)
run_config = DockerRun(dockerfile="Dockerfile")
with prefect.Flow("myflow") as flow:
results = task1()
... # Other tasks downstream
flow.register("test")
If I remove the result
parameter, everything runs fine.
If I decide to run this flow locally (changing storage, config and using flow.run()
instead of flow.register()
, I can save my results to GCS (using an environment variable for authentication)
I see two solutions at the moment:
• include the service account key when building my docker Image so I can reach it from inside the container the same way as locally
• GCSUpload
task to save my results (though it won't be possible to use these results as checkpoint), which in this case the GCP_CREDENTIALS secret from prefect Cloud should work ?