https://prefect.io logo
Powered by
Title
k

Keith

08/05/2022, 3:51 PM
Hi, trying to upload files generated from my extraction step to GCS and found the examples in 
prefect-gcp
to do it, but when I combine this with block information it seems like the info is not in the correct format. 
gcs_block = GCS.load("gcs-dev")

@flow()
def example_cloud_storage_upload_blob_from_file_flow():
    gcp_credentials = GcpCredentials(service_account_info=gcs_block.service_account_info)
    test_upload_file = "test_upload.txt"
    blob = cloud_storage_upload_blob_from_file(test_upload_path, gcs_block.bucket_path, "test_upload.txt", gcp_credentials)
    return blob
Running this produces the following error 
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for GcpCredentials
service_account_info
  JSON object must be str, bytes or bytearray (type=type_error.json)
a

Anna Geller

08/05/2022, 7:58 PM
it might be that the block is not configured correctly - could you check if you can use the block e.g. with a deployment as described here https://discourse.prefect.io/t/how-to-deploy-prefect-2-0-flows-to-gcp/1251 to confirm the block is working?
k

Keith

08/05/2022, 9:04 PM
Deployment files arrived in GCS
Figured out the issue, looks like you need 2 things 1. Call 
get_secret_value()
after the service_account_info is loaded from the block 2. Pass that object through 
json.dumps()
b/c the 
GcpCredentials
class is expecting a string. 
gcs_block = GCS.load("gcs-dev")

@flow()
def example_cloud_storage_upload_blob_from_file_flow():
    gcp_credentials = GcpCredentials(service_account_info=json.dumps(gcs_block.service_account_info.get_secret_value()))
    test_upload_file = "test_upload.txt"
    blob = cloud_storage_upload_blob_from_file(test_upload_file, gcs_block.bucket_path, "test_upload.txt", gcp_credentials)
    return blob
🙌 1
a

Anna Geller

08/05/2022, 11:20 PM
that makes sense since that value is a Secret field nice work figuring this out!
8 Views
#prefect-communityJoin Slack
Powered by