Hello I m getting the following error with DaskKubernetes en

Question

Hello I m getting the following error with DaskKubernetes environment It has to do with the environment not being able to read the secret in my prefect k8s agent ```Docker registry secret default docker does not exist for this tenant Creating Docker registry kubernetes secret from prefect Prefect Secret Failed to load and execute Flow s environment ClientError message No value found for the requested key locations line 2 column 9 path secret value extensions code INTERNAL SERVER ERROR exception errors message No value found for the requested key locations path secret value ``` Here is my flow ```from prefect import task Flow Parameter from prefect import Client from prefect environments storage import Docker from prefect environments import DaskKubernetesEnvironment import json task log stdout=True def say hello print Hello K8s format name return True task def print completed status print Success flow = Flow Simple Flow environment=DaskKubernetesEnvironment min workers=1 max workers=3 private registry=True docker secret= prefect storage=Docker image name= hello prefect registry url= print completed set upstream say hello flow=flow print completed bind status=say hello flow=flow flow register project name= Hello K8s ``` And here is the description of my agent s pod ```Name prefect agent 7fd7ff499 nzvlr Namespace default Priority 0 PriorityClassName lt none gt Node Start Time Thu 21 May 2020 15 51 33 0400 Labels app=prefect agent pod template hash=7fd7ff499 Annotations eks privileged Status Running IP 100 64 16 213 Controlled By ReplicaSet prefect agent 7fd7ff499 Containers agent Container ID f Image prefecthq prefect 0 11 2 python3 6 Image ID Port lt none gt Host Port lt none gt Command bin bash c Args prefect agent start kubernetes State Running Started Thu 21 May 2020 15 51 34 0400 Ready True Restart Count 0 Limits cpu 100m memory 128Mi Requests cpu 100m memory 128Mi Liveness http get http 8080 api health delay=40s timeout=1s period=40s success=1 failure=2 Environment PREFECT CLOUD AGENT AUTH TOKEN PREFECT CLOUD API NAMESPACE default IMAGE PULL SECRETS prefect PREFECT CLOUD AGENT LABELS JOB MEM REQUEST JOB MEM LIMIT JOB CPU REQUEST JOB CPU LIMIT PREFECT BACKEND cloud PREFECT CLOUD AGENT AGENT ADDRESS http 8080 Mounts var run secrets kubernetes io serviceaccount from default token 5nvmj ro Conditions Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes default token 5nvmj Type Secret a volume populated by a Secret SecretName default token 5nvmj Optional false QoS Class Guaranteed Node Selectors lt none gt Tolerations for 300s for 300s Events lt none gt ``` I ve verified that the secret named `prefect` pulls down my image from our company s registry

Dylan · Answer

Hi < Noah Nethery> What version of Prefect are you running If you re running off of `master` you might be encountering some trouble with this PR

Dylan · Answer

I think we re officially recommending that you should set imagePullSecrets as described here <https kubernetes io docs tasks configure pod container pull image private registry create a pod that uses your secret>

Noah Nethery · Answer

< Dylan> does this mean I will be using the custom configuration from a yaml in my Dask Kubernetes environment initialization

Noah Nethery · Answer

By the way I m on prefect version 0 11 2 according to pip

Noah Nethery · Answer

< Dylan> I also have an addendum Here is a new error I am getting on Prefect ```Unexpected error raised during flow run env Traceback most recent call last File usr local lib python3 7 site packages prefect environments execution dask k8s py line 277 in run flow worker pod = self populate worker spec yaml yaml obj=worker pod File usr local lib python3 7 site packages prefect environments execution dask k8s py line 555 in populate worker spec yaml env = yaml obj spec containers 0 env KeyError env ``` Here is my new flow ```from prefect import task Flow Parameter from prefect import Client from prefect environments storage import Docker from prefect environments import DaskKubernetesEnvironment task log stdout=True def say hello print Hello K8s format name return True task def print completed status print Success flow = Flow Simple Flow environment=DaskKubernetesEnvironment min workers=1 max workers=3 worker spec file= worker spec yaml storage=Docker image name= hello prefect registry url= print completed set upstream say hello flow=flow print completed bind status=say hello flow=flow flow register project name= Hello K8s ``` And here is my worker spec yaml ```kind Pod spec restartPolicy Never containers image hello prefect latest imagePullSecrets name prefect imagePullPolicy IfNotPresent args dask worker nthreads 2 no dashboard memory limit 6GB death timeout 60 name dask test resources limits cpu 2 memory 6G requests cpu 2 memory 6G```

Zachary Hughes · Answer

Hi Noah correct you ll want to specify custom worker and scheduler YAML if you re making use of image pull secrets For the new error you re getting it looks like your worker spec is missing an env section The example YAML below might be helpful ```spec restartPolicy Never containers image prefecthq prefect latest imagePullPolicy IfNotPresent args name dask worker env ```

Dylan · Answer

< Noah Nethery> Zachary s got you covered point up

Noah Nethery · Answer

< Zachary Hughes> Thank you I was able to get this implemented but now I m back to this error ```Failed to load and execute Flow s environment ClientError message No value found for the requested key locations line 2 column 9 path secret value extensions code INTERNAL SERVER ERROR exception errors message No value found for the requested key locations path secret value ``` I still don t understand since my k8s agent has my secret my Docker storage line in my flow has the secret name now and the worker spec yaml has the secret which I ve confirmed works since it s the pod from the job is able to pull the image no problem

Noah Nethery · Answer

```Name prefect job 456a6ef7 7qdds Namespace default Priority 0 PriorityClassName lt none gt Node Start Time Tue 26 May 2020 15 54 17 0400 Labels app=prefect job 456a6ef7 controller uid=af02435b 9f8a 11ea 998a 12e15268c2e9 flow run id=bdc7b4e5 d6f1 424a b74c 0b2d0e480720 identifier=456a6ef7 job name=prefect job 456a6ef7 Annotations eks privileged Status Failed IP 100 64 208 103 Controlled By Job prefect job 456a6ef7 Containers flow Container ID 7 Image Image ID docker pullable hello prefect sha256 7f32a4dbf900b87d85b9b29f778868edfea2d2bfe733e9b15b04feb7c1a29393 Port lt none gt Host Port lt none gt Command bin sh c Args prefect execute cloud flow State Terminated Reason Error Exit Code 1 Started Tue 26 May 2020 15 54 22 0400 Finished Tue 26 May 2020 15 54 27 0400 Ready False Restart Count 0 Limits cpu 100m Requests cpu 100m Environment PREFECT CLOUD API PREFECT CLOUD AUTH TOKEN PREFECT CONTEXT FLOW RUN ID bdc7b4e5 d6f1 424a b74c 0b2d0e480720 PREFECT CONTEXT FLOW ID 2f3c87ce 851d 4088 9af9 8051274d61d3 PREFECT CONTEXT NAMESPACE default PREFECT CLOUD AGENT LABELS PREFECT LOGGING LOG TO CLOUD true PREFECT CLOUD USE LOCAL SECRETS false PREFECT LOGGING LEVEL DEBUG PREFECT ENGINE FLOW RUNNER DEFAULT CLASS prefect engine cloud CloudFlowRunner PREFECT ENGINE TASK RUNNER DEFAULT CLASS prefect engine cloud CloudTaskRunner Mounts var run secrets kubernetes io serviceaccount from default token 5nvmj ro Conditions Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes default token 5nvmj Type Secret a volume populated by a Secret SecretName default token 5nvmj Optional false QoS Class Burstable Node Selectors lt none gt Tolerations for 300s for 300s Events Type Reason Age From Message Normal Scheduled 3m26s default scheduler Successfully assigned default prefect job 456a6ef7 7qdds to ip 100 64 223 131 ec2 internal Normal Pulling 3m25s kubelet ip 100 64 223 131 ec2 internal Pulling image hello prefect latest Normal Pulled 3m21s kubelet ip 100 64 223 131 ec2 internal Successfully pulled image hello prefect latest Normal Created 3m21s kubelet ip 100 64 223 131 ec2 internal Created container flow Normal Started 3m21s kubelet ip 100 64 223 131 ec2 internal Started container flow```

Zachary Hughes · Answer

Hmm The error you re posting looks like it s coming from the Cloud API Is there anywhere in your flow you ve set a dependency on a secret stored in Cloud

Noah Nethery · Answer

< Zachary Hughes> is there a way for me say to use local secrets instead of cloud ones

Zachary Hughes · Answer

< Noah Nethery> This PR should help you out as soon as it s merged In the meantime you could specify the secret via an environment variable or create a Cloud secret but I recognize that neither of those are particularly satisfying solutions