l

    Loic M

    1 year ago
    Hello everyone ! Sorry If this has already been answered here but my search did not return anything. I have trouble understanding authentication through google cloud to store my results with the help of prefect cloud secrets: Following this part of the documentation, I have setup a secret GCP_CREDENTIALS containing the JSON service account key. From what I understood, this should be enough to automatically authenticate my flows with the right credentials: however, when I run my docker Agent, my task fails with the following traceback:
    google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see <https://cloud.google.com/docs/authentication/getting-started|https://cloud.google.com/docs/authentication/getting->started
    Is there something I am missing there ? Should I include the google credentials when building my flow's docker image ?
    Josh

    Josh

    1 year ago
    I’ve bundled my credentials into the docker image, but I’d love to know if there’s a better way.
    nicholas

    nicholas

    1 year ago
    Hi @Loic M - can you give some more information about your setup? Are you using Prefect Cloud and trying to run tasks from the Task library?
    l

    Loic M

    1 year ago
    Hi @nicholas, I am using a cutsom task but try to save its result (a dictionary) into a
    GCSResult
    My code looks like this
    import prefect
    from prefect.engine.results import GCSResult
    from prefect.run_configs import DockerRun
    from prefect.storage import Docker
    
    @prefect.task(checkpoint=True, result =GCSResult(bucket="redacted"))
    def task1():
       ...
    
    storage = DockerStorage(...)
    run_config = DockerRun(dockerfile="Dockerfile")
    
    with prefect.Flow("myflow") as flow:
        results = task1()
        ... # Other tasks downstream
    
    flow.register("test")
    If I remove the
    result
    parameter, everything runs fine. If I decide to run this flow locally (changing storage, config and using
    flow.run()
    instead of
    flow.register()
    , I can save my results to GCS (using an environment variable for authentication) I see two solutions at the moment: • include the service account key when building my docker Image so I can reach it from inside the container the same way as locally • Create a
    GCSUpload
    task to save my results (though it won't be possible to use these results as checkpoint), which in this case the GCP_CREDENTIALS secret from prefect Cloud should work ?
    (Just tested this and was met with the same error as previously)