Do I understand correctly that new GCSResult will automatica Prefect Community #ask-community

Do I understand correctly that new GCSResult will automatically retrieve service account key stored in Prefect Cloud Secret called GCP_CREDENTIALS and use that for authentication? I have created the Secret called GCP_CREDENTIALS and stored in it the contents of SA key that I verified has access to specified GCS bucket, but in flow am getting

Copy code

Unexpected error: Forbidden("POST <https://storage.googleapis.com/upload/storage/v1/b>...

I don't see logging about having retrieved the GCP_CREDENTIALS as I might if I were explicitly using a Secret task, so it's a bit opaque

Dan DiPasquo

06/12/2020, 6:45 PM

Oh, does the SA key have to be of the REST API or client libraries form, not the Cloud Console or the gcloud command-line tool form? https://cloud.google.com/iam/docs/creating-managing-service-account-keys (uh, didn't even know there was a REST API or client libraries form)

Jenny

06/12/2020, 7:11 PM

Hi @Dan DiPasquo - I'm not so familiar with GCSResult but let me see if I can find an answer for you.

Jenny

06/12/2020, 7:37 PM

Hi @Dan DiPasquo - from my reading of that code, it looks like GCSResult should be calling GCP_CREDENTIALS but others in the team say they usually handle their GCP credentials separately. If you pass GCP_CREDENTIALS (or another secret with the same info) as a _credentials_secret_ argument to the GCSResult handler, does that work?

Jenny

06/12/2020, 7:38 PM

E.g.

Copy code

result_handler=GCSResultHandler(
       bucket="flow_result_handler",
       credentials_secret="XXX_GOOGLE_CLOUD_CREDENTIALS",
    )

Dan DiPasquo

06/12/2020, 8:12 PM

I can try it ASAP, but reading the code for GCSResult it makes a call _client = get_storage_client()_ and does not pass any arguments through to that function and so I am dubious

👍 1

josh

06/12/2020, 8:42 PM

Quick Q @Dan DiPasquo are you by any chance setting the secret on your flow’s storage with something like:

Copy code

flow.storage = Docker(..., secrets=["GCP_CREDENTIALS"])

Dan DiPasquo

06/12/2020, 9:16 PM

No, am not, did not see that noted in doc, I can give that at try if you suggests? https://docs.prefect.io/api/latest/environments/storage.html#docker

Dan DiPasquo

06/12/2020, 9:17 PM

Just have defined Secret called GCP_CREDENTIALS in Prefect Cloud with text value of the service credential .json file - does it need to be encoded in some way?

Dan DiPasquo

06/12/2020, 10:21 PM

Declaring secrets in the Docker storage instantiation results in an error:

Copy code

"Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/environments/execution/k8s/job.py", line 174, in run_flow
    secrets[secret.name] = secret.run()
AttributeError: 'str' object has no attribute 'run'

I'm trying to trace the code and don't follow that thread, storage.secrets stores a List[str] but it appears to be trying to used as a Secrets task?

Jenny

06/13/2020, 2:08 AM

Checking in here. Did you manage to get it working?

Dan DiPasquo

06/13/2020, 3:44 AM

Thanks for checking in @Jenny - alas no. Passing _credentials_secret="GCP_CREDENTIALS"_ into constructor results in a an error when I try to build the flow to register it -

Copy code

TypeError: __init__() got an unexpected keyword argument 'credentials_secret'

Not passing that in allows me to register it but then results in error during flow run seen in flow logs:

Copy code

Unexpected error: Forbidden("POST <https://storage.googleapis.com/upload/storage/v1/b/><<<bucket>>>>/o?uploadType=multipart: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>)")

Adding _secrets=["GCP_CREDENTIALS"],_ to Docker storage instantiation as josh suggested/asked results in error starting the flow seen in kubernetes logs:

Copy code

"Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/environments/execution/k8s/job.py", line 174, in run_flow
    secrets[secret.name] = secret.run()
AttributeError: 'str' object has no attribute 'run'

This behavior seems the same both for GCSUpload/GCSDownload Tasks and for GCSResult . So, I am not sure which way to go from here 😕

Dan DiPasquo

06/13/2020, 4:18 AM

I wrote a lot message with result of prefect.context.get("secrets", {}) call and got empty dict ; (I also logged prefect.context.get("config", {}) which gives me a nice dump of the config, so I know there's SOME context accessible, just no secrets) -- I do see that k8s.job.run_flo() is trying to

Copy code

## populate global secrets
secrets = prefect.context.get("secrets", {})
for secret in flow.storage.secrets:
    secrets[secret.name] = secret.run()

And that's where the error is when I add secrets=["GCP_CREDENTIALS"] to the docker storage intantiation -- flow.storage.secrets appears to contain a list of strings but here the code tries to run them as tasks --

Jenny

06/13/2020, 2:20 PM

Thanks for following up Dan. I'm going to see if anyone in the wider team has any ideas.

Chris White

06/13/2020, 4:51 PM

Thanks @Jenny for bringing this to my attention! @Dan DiPasquo, it appears you’ve found a small bug in the k8s job environment (and coincidentally the dask k8s environment as well). In short, adding the JSON directly through the Cloud UI + declaring the secret on your Docker storage will behave as you expect once this bug is fixed. PR for the fix here: https://github.com/PrefectHQ/prefect/pull/2780

👍 1

Dan DiPasquo

06/14/2020, 12:42 AM

Thanks for tackling that bug @Chris White , good news that there's a fix forthcoming -

Dan DiPasquo

06/16/2020, 11:38 PM

Just confirming that this fixes issue I was having; when building off prefect master, I'm now able to set GCP_CREDENTIALS as a Secret in Prefect Cloud, declare _secrets=["GCP_CREDENTIALS"]_ and GCSResults authenticate as expected

Dan DiPasquo

06/16/2020, 11:38 PM

ETA on 0.11.6 release?

josh

06/17/2020, 2:21 AM

0.12.0

will be out this week!

🚀 2

3 Views

Open in Slack

Previous Next