https://prefect.io logo
Title
d

Dan DiPasquo

06/12/2020, 6:39 PM
Do I understand correctly that new GCSResult will automatically retrieve service account key stored in Prefect Cloud Secret called GCP_CREDENTIALS and use that for authentication? I have created the Secret called GCP_CREDENTIALS and stored in it the contents of SA key that I verified has access to specified GCS bucket, but in flow am getting
Unexpected error: Forbidden("POST <https://storage.googleapis.com/upload/storage/v1/b>...
I don't see logging about having retrieved the GCP_CREDENTIALS as I might if I were explicitly using a Secret task, so it's a bit opaque
Oh, does the SA key have to be of the REST API or client libraries form, not the Cloud Console or the gcloud command-line tool form? https://cloud.google.com/iam/docs/creating-managing-service-account-keys (uh, didn't even know there was a REST API or client libraries form)
j

Jenny

06/12/2020, 7:11 PM
Hi @Dan DiPasquo - I'm not so familiar with GCSResult but let me see if I can find an answer for you.
Hi @Dan DiPasquo - from my reading of that code, it looks like GCSResult should be calling GCP_CREDENTIALS but others in the team say they usually handle their GCP credentials separately. If you pass GCP_CREDENTIALS (or another secret with the same info) as a _credentials_secret_ argument to the GCSResult handler, does that work?
E.g.
result_handler=GCSResultHandler(
       bucket="flow_result_handler",
       credentials_secret="XXX_GOOGLE_CLOUD_CREDENTIALS",
    )
d

Dan DiPasquo

06/12/2020, 8:12 PM
I can try it ASAP, but reading the code for GCSResult it makes a call _client = get_storage_client()_ and does not pass any arguments through to that function and so I am dubious
👍 1
j

josh

06/12/2020, 8:42 PM
Quick Q @Dan DiPasquo are you by any chance setting the secret on your flow’s storage with something like:
flow.storage = Docker(..., secrets=["GCP_CREDENTIALS"])
d

Dan DiPasquo

06/12/2020, 9:16 PM
No, am not, did not see that noted in doc, I can give that at try if you suggests? https://docs.prefect.io/api/latest/environments/storage.html#docker
Just have defined Secret called GCP_CREDENTIALS in Prefect Cloud with text value of the service credential .json file - does it need to be encoded in some way?
Declaring secrets in the Docker storage instantiation results in an error:
"Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/environments/execution/k8s/job.py", line 174, in run_flow
    secrets[secret.name] = secret.run()
AttributeError: 'str' object has no attribute 'run'
I'm trying to trace the code and don't follow that thread, storage.secrets stores a List[str] but it appears to be trying to used as a Secrets task?
j

Jenny

06/13/2020, 2:08 AM
Checking in here. Did you manage to get it working?
d

Dan DiPasquo

06/13/2020, 3:44 AM
Thanks for checking in @Jenny - alas no. Passing _credentials_secret="GCP_CREDENTIALS"_ into constructor results in a an error when I try to build the flow to register it -
TypeError: __init__() got an unexpected keyword argument 'credentials_secret'
Not passing that in allows me to register it but then results in error during flow run seen in flow logs:
Unexpected error: Forbidden("POST <https://storage.googleapis.com/upload/storage/v1/b/><<<bucket>>>>/o?uploadType=multipart: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>)")
Adding _secrets=["GCP_CREDENTIALS"],_ to Docker storage instantiation as josh suggested/asked results in error starting the flow seen in kubernetes logs:
"Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/environments/execution/k8s/job.py", line 174, in run_flow
    secrets[secret.name] = secret.run()
AttributeError: 'str' object has no attribute 'run'
This behavior seems the same both for GCSUpload/GCSDownload Tasks and for GCSResult . So, I am not sure which way to go from here 😕
I wrote a lot message with result of prefect.context.get("secrets", {}) call and got empty dict ; (I also logged prefect.context.get("config", {}) which gives me a nice dump of the config, so I know there's SOME context accessible, just no secrets) -- I do see that k8s.job.run_flo() is trying to
## populate global secrets
secrets = prefect.context.get("secrets", {})
for secret in flow.storage.secrets:
    secrets[secret.name] = secret.run()
And that's where the error is when I add secrets=["GCP_CREDENTIALS"] to the docker storage intantiation -- flow.storage.secrets appears to contain a list of strings but here the code tries to run them as tasks --
j

Jenny

06/13/2020, 2:20 PM
Thanks for following up Dan. I'm going to see if anyone in the wider team has any ideas.
c

Chris White

06/13/2020, 4:51 PM
Thanks @Jenny for bringing this to my attention! @Dan DiPasquo, it appears you’ve found a small bug in the k8s job environment (and coincidentally the dask k8s environment as well). In short, adding the JSON directly through the Cloud UI + declaring the secret on your Docker storage will behave as you expect once this bug is fixed. PR for the fix here: https://github.com/PrefectHQ/prefect/pull/2780
👍 1
d

Dan DiPasquo

06/14/2020, 12:42 AM
Thanks for tackling that bug @Chris White , good news that there's a fix forthcoming -
Just confirming that this fixes issue I was having; when building off prefect master, I'm now able to set GCP_CREDENTIALS as a Secret in Prefect Cloud, declare _secrets=["GCP_CREDENTIALS"]_ and GCSResults authenticate as expected
ETA on 0.11.6 release?
j

josh

06/17/2020, 2:21 AM
0.12.0
will be out this week!
🚀 2