Taylor Harless

    Taylor Harless

    6 months ago
    I'm struggling to implement the GCP_CREDENTIALS default secret feature in a local Prefect flow. • I've set an environmental variable PREFECT__CONTEXT__SECRETS_GCP_CREDENTIALS pointing to a full local path for the GCP credentials .json downloaded from the GCP project to which I'd like to upload some files. • I've run the following:
    from prefect import Flow
    from prefect.tasks.gcp.storage import GCSUpload
    
    with Flow("google-cloud-test") as flow:
    
        GCSUpload(bucket="test-upload", create_bucket=True)(
            data="test-file.csv", credentials="GCP_CREDENTIALS"
        )
    
    flow.run()
    and received an error:
    AttributeError: 'str' object has no attribute 'keys'
    . I've incorporated the feedback from a similar error discussion recently, but can't figure out what the issue is. Any help is much appreciated.
    Kevin Kho

    Kevin Kho

    6 months ago
    Hi @Taylor Harless, I think Google convention and Prefect convention are different here. Google convention for this is an env variable named
    GOOGLE_APPLICATION_CREDENTIALS
    that points to the json. The
    GCP_CREDENTIALS
    though contains the service account information. It is fed into line 29 here. If the credentials aren’t there, it goes to line 33 where it uses the default
    GOOGLE_APPLICATION_CREDENTIALS
    Taylor Harless

    Taylor Harless

    6 months ago
    Hi Kevin, thanks for the quick response. Yes, I can see that those are two different conventions. I've tried defining the GCP_CREDENTIALS env variable with the contents of the json, but it's too large/long to fully add. I'll could try cherry picking a few of the key/value pairs from the .json to add to the GCP_CREDENTIALS variable to see if that solves it. Any ideas on other methods I should try?
    Kevin Kho

    Kevin Kho

    6 months ago
    Man I swear their docs had an example somewhere but I can’t fine the page. What error do you get when you do that? You could have just left the
    GOOGLE_APPLICATION_CREDENTIALS
    env variable too and that would have worked as long as your execution environment has the file
    I would say fiddle with what makes the
    Credentials.from_service_account_info
    happy. You can test this outside of Prefect by just importing and creating the connection. And then whatever works there will work has the secret value
    Taylor Harless

    Taylor Harless

    6 months ago
    I get the same error AttributeError when I leave: 1. remove the
    credentials
    attribute definition, or 2. update the
    PREFECT__CONTEXT__SECRETS_GCP_CREDENTIALS
    env variable to contain the partial dictionary inside the GCP credentials json. I appreciate all the good feedback, and on a weekend no less - I'll look at the Credentials class you mentioned at some point tomorrow and report back here.
    Kevin Kho

    Kevin Kho

    6 months ago
    I can try this more tonight
    Anna Geller

    Anna Geller

    6 months ago
    What's your Prefect setup @Taylor Harless ? Do you use Prefect Cloud or Server? What agent type and flow storage do you use?
    Since I don't know which agent you use, let me explain it for a Local and Docker agents. Docstring in this example explains how you can set GCP credentials. The easiest is to use the Google convention, as Kevin explained. To set this up, create a service account JSON file as explained here, then in your execution environment (e.g. on your agent) set this env variable:
    export GOOGLE_APPLICATION_CREDENTIALS="/Users/yourname/path/to/your/repo/packaging-prefect-flows/gcs_sa.json"
    But note that this variable is just a path to the JSON file, rather than its content with the actual secret. For uploading things to GCS, you can set this up directly on your agent as follows: • for a Docker agent - the command below will make sure to mount the credentials JSON file to each flow run container and set the environment variable
    GOOGLE_APPLICATION_CREDENTIALS
    to ensure that GCP modules such as
    GCSUpload
    can find where this mounted file is located within your flow run Docker container (i.e. your execution environment)
    prefect agent docker start --label AGENT_LABEL --volume /Users/anna/repos/packaging-prefect-flows/gcs_sa.json:/opt/prefect/gcs_sa.json --env GOOGLE_APPLICATION_CREDENTIALS="/opt/prefect/gcs_sa.json"
    • for a Local agent, you can do the same but you don't have to mount a volume since everything runs locally on the same VM
    prefect agent local start --label AGENT_LABEL --env GOOGLE_APPLICATION_CREDENTIALS="/Users/anna/repos/packaging-prefect-flows/gcs_sa.json"
    Adjust the path to where you stored your JSON file and you should be good to go.
    Kevin Kho

    Kevin Kho

    6 months ago
    I understand what you are saying now. So you can create a Secret by setting it with the
    os
    library before you import Prefect. I did that here to understand the format and everything works perfectly:
    import json
    import os
    service_account_info = json.load(open('test.json'))
    os.environ["PREFECT__CONTEXT__SECRETS__GCP_CREDENTIALS"] = json.dumps(service_account_info)
    
    from google.oauth2.service_account import Credentials
    from google.cloud import storage
    from prefect.client import Secret
    
    Client = getattr(storage, "Client")
    service_account_info = Secret("GCP_CREDENTIALS").get()
    credentials = Credentials.from_service_account_info(
        service_account_info)
    project = credentials.project_id
    client = Client(project=project, credentials=credentials)
    buckets = client.list_buckets()
    
    for bucket in buckets:
        print(bucket.name)
    The problem then becomes how to set the credentials as an env variable so that it’s ready for the script. I can’t paste the json because I hit some sort of character limit. It is also a massive pain to get it correctly in the
    config.toml
    . I haven’t figured out how. I feel this is not a problem for Cloud users because Cloud users can just set the secret in the UI and you can just paste the entire thing in. For you, if you are on server, I recommend just using
    GOOGLE_APPLICATION_CREDENTIALS
    Taylor Harless

    Taylor Harless

    6 months ago
    @Anna Geller and @Kevin Kho, thank you both so much for the detailed responses over the weekend! I haven't been able to return to this particular project yet, but when I do, I'll report back with the solution that works for future reference.