Taylor Harless03/19/2022, 5:29 PM
and received an error:
from prefect import Flow from prefect.tasks.gcp.storage import GCSUpload with Flow("google-cloud-test") as flow: GCSUpload(bucket="test-upload", create_bucket=True)( data="test-file.csv", credentials="GCP_CREDENTIALS" ) flow.run()
. I've incorporated the feedback from a similar error discussion recently, but can't figure out what the issue is. Any help is much appreciated.
AttributeError: 'str' object has no attribute 'keys'
Kevin Kho03/19/2022, 5:33 PM
that points to the json. The
though contains the service account information. It is fed into line 29 here. If the credentials aren’t there, it goes to line 33 where it uses the default
Taylor Harless03/19/2022, 5:48 PM
Kevin Kho03/19/2022, 6:27 PM
env variable too and that would have worked as long as your execution environment has the file
happy. You can test this outside of Prefect by just importing and creating the connection. And then whatever works there will work has the secret value
Taylor Harless03/19/2022, 6:43 PM
attribute definition, or 2. update the
env variable to contain the partial dictionary inside the GCP credentials json. I appreciate all the good feedback, and on a weekend no less - I'll look at the Credentials class you mentioned at some point tomorrow and report back here.
Kevin Kho03/19/2022, 6:52 PM
Anna Geller03/19/2022, 9:51 PM
But note that this variable is just a path to the JSON file, rather than its content with the actual secret. For uploading things to GCS, you can set this up directly on your agent as follows: • for a Docker agent - the command below will make sure to mount the credentials JSON file to each flow run container and set the environment variable
to ensure that GCP modules such as
can find where this mounted file is located within your flow run Docker container (i.e. your execution environment)
• for a Local agent, you can do the same but you don't have to mount a volume since everything runs locally on the same VM
prefect agent docker start --label AGENT_LABEL --volume /Users/anna/repos/packaging-prefect-flows/gcs_sa.json:/opt/prefect/gcs_sa.json --env GOOGLE_APPLICATION_CREDENTIALS="/opt/prefect/gcs_sa.json"
Adjust the path to where you stored your JSON file and you should be good to go.
prefect agent local start --label AGENT_LABEL --env GOOGLE_APPLICATION_CREDENTIALS="/Users/anna/repos/packaging-prefect-flows/gcs_sa.json"
Kevin Kho03/19/2022, 11:58 PM
library before you import Prefect. I did that here to understand the format and everything works perfectly:
The problem then becomes how to set the credentials as an env variable so that it’s ready for the script. I can’t paste the json because I hit some sort of character limit. It is also a massive pain to get it correctly in the
import json import os service_account_info = json.load(open('test.json')) os.environ["PREFECT__CONTEXT__SECRETS__GCP_CREDENTIALS"] = json.dumps(service_account_info) from google.oauth2.service_account import Credentials from google.cloud import storage from prefect.client import Secret Client = getattr(storage, "Client") service_account_info = Secret("GCP_CREDENTIALS").get() credentials = Credentials.from_service_account_info( service_account_info) project = credentials.project_id client = Client(project=project, credentials=credentials) buckets = client.list_buckets() for bucket in buckets: print(bucket.name)
. I haven’t figured out how. I feel this is not a problem for Cloud users because Cloud users can just set the secret in the UI and you can just paste the entire thing in. For you, if you are on server, I recommend just using
Taylor Harless03/21/2022, 8:09 PM