https://prefect.io logo
Channels
Powered by
l

Loic M

01/15/2021, 5:36 PM
Hello everyone ! Sorry If this has already been answered here but my search did not return anything. I have trouble understanding authentication through google cloud to store my results with the help of prefect cloud secrets: Following this part of the documentation, I have setup a secret GCP_CREDENTIALS containing the JSON service account key. From what I understood, this should be enough to automatically authenticate my flows with the right credentials: however, when I run my docker Agent, my task fails with the following traceback: 
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see <https://cloud.google.com/docs/authentication/getting-started|https://cloud.google.com/docs/authentication/getting->started
Is there something I am missing there ? Should I include the google credentials when building my flow's docker image ?
j

Josh

01/15/2021, 5:39 PM
I’ve bundled my credentials into the docker image, but I’d love to know if there’s a better way.
n

nicholas

01/15/2021, 6:10 PM
Hi @Loic M - can you give some more information about your setup? Are you using Prefect Cloud and trying to run tasks from the Task library?
l

Loic M

01/16/2021, 1:32 PM
Hi @nicholas, I am using a cutsom task but try to save its result (a dictionary) into a 
GCSResult
My code looks like this
Copy code
import prefect
from prefect.engine.results import GCSResult
from prefect.run_configs import DockerRun
from prefect.storage import Docker

@prefect.task(checkpoint=True, result =GCSResult(bucket="redacted"))
def task1():
   ...

storage = DockerStorage(...)
run_config = DockerRun(dockerfile="Dockerfile")

with prefect.Flow("myflow") as flow:
    results = task1()
    ... # Other tasks downstream

flow.register("test")
If I remove the 
result
parameter, everything runs fine. If I decide to run this flow locally (changing storage, config and using 
flow.run()
instead of 
flow.register()
, I can save my results to GCS (using an environment variable for authentication) I see two solutions at the moment: • include the service account key when building my docker Image so I can reach it from inside the container the same way as locally • Create a 
GCSUpload
task to save my results (though it won't be possible to use these results as checkpoint), which in this case the GCP_CREDENTIALS secret from prefect Cloud should work ? (Just tested this and was met with the same error as previously)
2 Views
Powered by