Taylor Harless
03/19/2022, 5:29 PMfrom prefect import Flow
from prefect.tasks.gcp.storage import GCSUpload
with Flow("google-cloud-test") as flow:
GCSUpload(bucket="test-upload", create_bucket=True)(
data="test-file.csv", credentials="GCP_CREDENTIALS"
)
flow.run()
and received an error: AttributeError: 'str' object has no attribute 'keys'
. I've incorporated the feedback from a similar error discussion recently, but can't figure out what the issue is. Any help is much appreciated.Kevin Kho
GOOGLE_APPLICATION_CREDENTIALS
that points to the json. The GCP_CREDENTIALS
though contains the service account information. It is fed into line 29 here. If the credentials aren’t there, it goes to line 33 where it uses the default GOOGLE_APPLICATION_CREDENTIALS
Taylor Harless
03/19/2022, 5:48 PMKevin Kho
GOOGLE_APPLICATION_CREDENTIALS
env variable too and that would have worked as long as your execution environment has the fileCredentials.from_service_account_info
happy. You can test this outside of Prefect by just importing and creating the connection. And then whatever works there will work has the secret valueTaylor Harless
03/19/2022, 6:43 PMcredentials
attribute definition, or 2. update the PREFECT__CONTEXT__SECRETS_GCP_CREDENTIALS
env variable to contain the partial dictionary inside the GCP credentials json. I appreciate all the good feedback, and on a weekend no less - I'll look at the Credentials class you mentioned at some point tomorrow and report back here.Kevin Kho
Anna Geller
export GOOGLE_APPLICATION_CREDENTIALS="/Users/yourname/path/to/your/repo/packaging-prefect-flows/gcs_sa.json"
But note that this variable is just a path to the JSON file, rather than its content with the actual secret.
For uploading things to GCS, you can set this up directly on your agent as follows:
• for a Docker agent - the command below will make sure to mount the credentials JSON file to each flow run container and set the environment variable GOOGLE_APPLICATION_CREDENTIALS
to ensure that GCP modules such as GCSUpload
can find where this mounted file is located within your flow run Docker container (i.e. your execution environment)
prefect agent docker start --label AGENT_LABEL --volume /Users/anna/repos/packaging-prefect-flows/gcs_sa.json:/opt/prefect/gcs_sa.json --env GOOGLE_APPLICATION_CREDENTIALS="/opt/prefect/gcs_sa.json"
• for a Local agent, you can do the same but you don't have to mount a volume since everything runs locally on the same VM
prefect agent local start --label AGENT_LABEL --env GOOGLE_APPLICATION_CREDENTIALS="/Users/anna/repos/packaging-prefect-flows/gcs_sa.json"
Adjust the path to where you stored your JSON file and you should be good to go.Kevin Kho
os
library before you import Prefect. I did that here to understand the format and everything works perfectly:
import json
import os
service_account_info = json.load(open('test.json'))
os.environ["PREFECT__CONTEXT__SECRETS__GCP_CREDENTIALS"] = json.dumps(service_account_info)
from google.oauth2.service_account import Credentials
from google.cloud import storage
from prefect.client import Secret
Client = getattr(storage, "Client")
service_account_info = Secret("GCP_CREDENTIALS").get()
credentials = Credentials.from_service_account_info(
service_account_info)
project = credentials.project_id
client = Client(project=project, credentials=credentials)
buckets = client.list_buckets()
for bucket in buckets:
print(bucket.name)
The problem then becomes how to set the credentials as an env variable so that it’s ready for the script. I can’t paste the json because I hit some sort of character limit.
It is also a massive pain to get it correctly in the config.toml
. I haven’t figured out how. I feel this is not a problem for Cloud users because Cloud users can just set the secret in the UI and you can just paste the entire thing in. For you, if you are on server, I recommend just using GOOGLE_APPLICATION_CREDENTIALS
Taylor Harless
03/21/2022, 8:09 PM