Dan DiPasquo
06/12/2020, 6:39 PMUnexpected error: Forbidden("POST <https://storage.googleapis.com/upload/storage/v1/b>...
I don't see logging about having retrieved the GCP_CREDENTIALS as I might if I were explicitly using a Secret task, so it's a bit opaqueJenny
06/12/2020, 7:11 PMresult_handler=GCSResultHandler(
bucket="flow_result_handler",
credentials_secret="XXX_GOOGLE_CLOUD_CREDENTIALS",
)
Dan DiPasquo
06/12/2020, 8:12 PMjosh
06/12/2020, 8:42 PMflow.storage = Docker(..., secrets=["GCP_CREDENTIALS"])
Dan DiPasquo
06/12/2020, 9:16 PM"Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/environments/execution/k8s/job.py", line 174, in run_flow
secrets[secret.name] = secret.run()
AttributeError: 'str' object has no attribute 'run'
I'm trying to trace the code and don't follow that thread, storage.secrets stores a List[str] but it appears to be trying to used as a Secrets task?Jenny
06/13/2020, 2:08 AMDan DiPasquo
06/13/2020, 3:44 AMTypeError: __init__() got an unexpected keyword argument 'credentials_secret'
Not passing that in allows me to register it but then results in error during flow run seen in flow logs:
Unexpected error: Forbidden("POST <https://storage.googleapis.com/upload/storage/v1/b/><<<bucket>>>>/o?uploadType=multipart: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>)")
Adding _secrets=["GCP_CREDENTIALS"],_ to Docker storage instantiation as josh suggested/asked results in error starting the flow seen in kubernetes logs:
"Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/environments/execution/k8s/job.py", line 174, in run_flow
secrets[secret.name] = secret.run()
AttributeError: 'str' object has no attribute 'run'
This behavior seems the same both for GCSUpload/GCSDownload Tasks and for GCSResult . So, I am not sure which way to go from here 😕## populate global secrets
secrets = prefect.context.get("secrets", {})
for secret in flow.storage.secrets:
secrets[secret.name] = secret.run()
And that's where the error is when I add secrets=["GCP_CREDENTIALS"] to the docker storage intantiation -- flow.storage.secrets appears to contain a list of strings but here the code tries to run them as tasks --Jenny
06/13/2020, 2:20 PMChris White
06/13/2020, 4:51 PMDan DiPasquo
06/14/2020, 12:42 AMjosh
06/17/2020, 2:21 AM0.12.0
will be out this week!