https://prefect.io logo
Channels
Powered by
t

Taylor Harless

03/19/2022, 5:29 PM
I'm struggling to implement the GCP_CREDENTIALS default secret feature in a local Prefect flow. • I've set an environmental variable PREFECT__CONTEXT__SECRETS_GCP_CREDENTIALS pointing to a full local path for the GCP credentials .json downloaded from the GCP project to which I'd like to upload some files. • I've run the following:
Copy code
from prefect import Flow
from prefect.tasks.gcp.storage import GCSUpload

with Flow("google-cloud-test") as flow:

    GCSUpload(bucket="test-upload", create_bucket=True)(
        data="test-file.csv", credentials="GCP_CREDENTIALS"
    )

flow.run()
and received an error: 
AttributeError: 'str' object has no attribute 'keys'
. I've incorporated the feedback from a similar error discussion recently, but can't figure out what the issue is. Any help is much appreciated.
k

Kevin Kho

03/19/2022, 5:33 PM
Hi @Taylor Harless, I think Google convention and Prefect convention are different here. Google convention for this is an env variable named 
GOOGLE_APPLICATION_CREDENTIALS
that points to the json. The 
GCP_CREDENTIALS
though contains the service account information. It is fed into line 29 here. If the credentials aren’t there, it goes to line 33 where it uses the default 
GOOGLE_APPLICATION_CREDENTIALS
t

Taylor Harless

03/19/2022, 5:48 PM
Hi Kevin, thanks for the quick response. Yes, I can see that those are two different conventions. I've tried defining the GCP_CREDENTIALS env variable with the contents of the json, but it's too large/long to fully add. I'll could try cherry picking a few of the key/value pairs from the .json to add to the GCP_CREDENTIALS variable to see if that solves it. Any ideas on other methods I should try?
k

Kevin Kho

03/19/2022, 6:27 PM
Man I swear their docs had an example somewhere but I can’t fine the page. What error do you get when you do that? You could have just left the 
GOOGLE_APPLICATION_CREDENTIALS
env variable too and that would have worked as long as your execution environment has the file
I would say fiddle with what makes the 
Credentials.from_service_account_info
happy. You can test this outside of Prefect by just importing and creating the connection. And then whatever works there will work has the secret value
t

Taylor Harless

03/19/2022, 6:43 PM
I get the same error AttributeError when I leave: 1. remove the 
credentials
attribute definition, or 2. update the 
PREFECT__CONTEXT__SECRETS_GCP_CREDENTIALS
env variable to contain the partial dictionary inside the GCP credentials json. I appreciate all the good feedback, and on a weekend no less - I'll look at the Credentials class you mentioned at some point tomorrow and report back here.
k

Kevin Kho

03/19/2022, 6:52 PM
I can try this more tonight
a

Anna Geller

03/19/2022, 9:51 PM
What's your Prefect setup @Taylor Harless ? Do you use Prefect Cloud or Server? What agent type and flow storage do you use?
Since I don't know which agent you use, let me explain it for a Local and Docker agents. Docstring in this example explains how you can set GCP credentials. The easiest is to use the Google convention, as Kevin explained. To set this up, create a service account JSON file as explained here, then in your execution environment (e.g. on your agent) set this env variable:
Copy code
export GOOGLE_APPLICATION_CREDENTIALS="/Users/yourname/path/to/your/repo/packaging-prefect-flows/gcs_sa.json"
But note that this variable is just a path to the JSON file, rather than its content with the actual secret. For uploading things to GCS, you can set this up directly on your agent as follows: • for a Docker agent - the command below will make sure to mount the credentials JSON file to each flow run container and set the environment variable 
GOOGLE_APPLICATION_CREDENTIALS
to ensure that GCP modules such as 
GCSUpload
can find where this mounted file is located within your flow run Docker container (i.e. your execution environment)
Copy code
prefect agent docker start --label AGENT_LABEL --volume /Users/anna/repos/packaging-prefect-flows/gcs_sa.json:/opt/prefect/gcs_sa.json --env GOOGLE_APPLICATION_CREDENTIALS="/opt/prefect/gcs_sa.json"
• for a Local agent, you can do the same but you don't have to mount a volume since everything runs locally on the same VM
Copy code
prefect agent local start --label AGENT_LABEL --env GOOGLE_APPLICATION_CREDENTIALS="/Users/anna/repos/packaging-prefect-flows/gcs_sa.json"
Adjust the path to where you stored your JSON file and you should be good to go.
k

Kevin Kho

03/19/2022, 11:58 PM
I understand what you are saying now. So you can create a Secret by setting it with the 
os
library before you import Prefect. I did that here to understand the format and everything works perfectly:
Copy code
import json
import os
service_account_info = json.load(open('test.json'))
os.environ["PREFECT__CONTEXT__SECRETS__GCP_CREDENTIALS"] = json.dumps(service_account_info)

from google.oauth2.service_account import Credentials
from google.cloud import storage
from prefect.client import Secret

Client = getattr(storage, "Client")
service_account_info = Secret("GCP_CREDENTIALS").get()
credentials = Credentials.from_service_account_info(
    service_account_info)
project = credentials.project_id
client = Client(project=project, credentials=credentials)
buckets = client.list_buckets()

for bucket in buckets:
    print(bucket.name)
The problem then becomes how to set the credentials as an env variable so that it’s ready for the script. I can’t paste the json because I hit some sort of character limit. It is also a massive pain to get it correctly in the 
config.toml
. I haven’t figured out how. I feel this is not a problem for Cloud users because Cloud users can just set the secret in the UI and you can just paste the entire thing in. For you, if you are on server, I recommend just using 
GOOGLE_APPLICATION_CREDENTIALS
upvote 1
t

Taylor Harless

03/21/2022, 8:09 PM
@Anna Geller and @Kevin Kho, thank you both so much for the detailed responses over the weekend! I haven't been able to return to this particular project yet, but when I do, I'll report back with the solution that works for future reference.
🙌 1
👍 1
19 Views
Powered by