Do I understand correctly that new GCSResult will automatica

Question

Do I understand correctly that new GCSResult will automatically retrieve service account key stored in Prefect Cloud Secret called GCP CREDENTIALS and use that for authentication I have created the Secret called GCP CREDENTIALS and stored in it the contents of SA key that I verified has access to specified GCS bucket but in flow am getting ```Unexpected error Forbidden POST <https storage googleapis com upload storage v1 b> ``` I don t see logging about having retrieved the GCP CREDENTIALS as I might if I were explicitly using a Secret task so it s a bit opaque

Dan DiPasquo · Answer

Oh does the SA key have to be of the REST API or client libraries form not the Cloud Console or the gcloud command line tool form <https cloud google com iam docs creating managing service account keys> uh didn t even know there was a REST API or client libraries form

Jenny · Answer

Hi < Dan DiPasquo> I m not so familiar with GCSResult but let me see if I can find an answer for you

Jenny · Answer

Hi < Dan DiPasquo> from my reading of that code it looks like GCSResult should be calling GCP CREDENTIALS but others in the team say they usually handle their GCP credentials separately If you pass GCP CREDENTIALS or another secret with the same info as a credentials secret argument to the GCSResult handler does that work

Jenny · Answer

E g ```result handler=GCSResultHandler bucket= flow result handler credentials secret= XXX GOOGLE CLOUD CREDENTIALS ```

Dan DiPasquo · Answer

I can try it ASAP but reading the code for GCSResult it makes a call client = get storage client and does not pass any arguments through to that function and so I am dubious

josh · Answer

Quick Q < Dan DiPasquo> are you by any chance setting the secret on your flow s storage with something like ```flow storage = Docker secrets= GCP CREDENTIALS ```

Dan DiPasquo · Answer

No am not did not see that noted in doc I can give that at try if you suggests <https docs prefect io api latest environments storage html docker>

Dan DiPasquo · Answer

Just have defined Secret called GCP CREDENTIALS in Prefect Cloud with text value of the service credential json file does it need to be encoded in some way

Dan DiPasquo · Answer

Declaring secrets in the Docker storage instantiation results in an error ``` Traceback most recent call last File usr local lib python3 8 site packages prefect environments execution k8s job py line 174 in run flow secrets secret name = secret run AttributeError str object has no attribute run ``` I m trying to trace the code and don t follow that thread storage secrets stores a List str but it appears to be trying to used as a Secrets task

Jenny · Answer

Checking in here Did you manage to get it working

Dan DiPasquo · Answer

Thanks for checking in < Jenny> alas no Passing credentials secret= GCP CREDENTIALS into constructor results in a an error when I try to build the flow to register it ```TypeError init got an unexpected keyword argument credentials secret ``` Not passing that in allows me to register it but then results in error during flow run seen in flow logs ```Unexpected error Forbidden POST lt lt lt bucket gt gt gt gt o uploadType=multipart Request failed with status code 403 Expected one of lt HTTPStatus OK 200 gt ``` Adding secrets= GCP CREDENTIALS to Docker storage instantiation as josh suggested asked results in error starting the flow seen in kubernetes logs ``` Traceback most recent call last File usr local lib python3 8 site packages prefect environments execution k8s job py line 174 in run flow secrets secret name = secret run AttributeError str object has no attribute run ``` This behavior seems the same both for GCSUpload GCSDownload Tasks and for GCSResult So I am not sure which way to go from here confused

Dan DiPasquo · Answer

I wrote a lot message with result of prefect context get secrets call and got empty dict I also logged prefect context get config which gives me a nice dump of the config so I know there s SOME context accessible just no secrets I do see that k8s job run flo is trying to ``` populate global secrets secrets = prefect context get secrets for secret in flow storage secrets secrets secret name = secret run ``` And that s where the error is when I add secrets= GCP CREDENTIALS to the docker storage intantiation flow storage secrets appears to contain a list of strings but here the code tries to run them as tasks

Jenny · Answer

Thanks for following up Dan I m going to see if anyone in the wider team has any ideas

Chris White · Answer

Thanks < Jenny> for bringing this to my attention < Dan DiPasquo> it appears you ve found a small bug in the k8s job environment and coincidentally the dask k8s environment as well In short adding the JSON directly through the Cloud UI + declaring the secret on your Docker storage will behave as you expect once this bug is fixed PR for the fix here

Dan DiPasquo · Answer

Thanks for tackling that bug < Chris White> good news that there s a fix forthcoming

Dan DiPasquo · Answer

Just confirming that this fixes issue I was having when building off prefect master I m now able to set GCP CREDENTIALS as a Secret in Prefect Cloud declare secrets= GCP CREDENTIALS and GCSResults authenticate as expected

Dan DiPasquo · Answer

ETA on 0 11 6 release

josh · Answer

`0 12 0` will be out this week