Hello again we ve been able to successfully run our Dask wor

Question

Hello again we ve been able to successfully run our Dask workers in our kubernetes EKS cluster by installing a kubernetes agent and although our prefect cloud logs status displays that the flow has been executed successfully we still got a log entry at the very end stating ```response body kind Status apiVersion v1 metadata status Failure message services is forbidden User system serviceaccount default default cannot list resource services in API group in the namespace default reason Forbidden details kind services code 403 ``` PS We can see sucessful response bodies but should we ignore this last entry is it intended Thanks in advance

Laura Lorenz (she/her) · Answer

Hi < bruno corucho> This looks like a EKS log to me at first glance and implies something is up with your RBAC at some point when kubernetes is trying to do some kubernetes thing FWIW I haven t seen this error in our kubernetes runs myself What environment are you using And what is the last task runner or flow runner log before this EKS log you see so I can triangulate more precisely where we are in the pipeline and see if I can figure out what kubernetes thing kubernetes is trying to do haha

bruno.corucho · Answer

Hi again < Laura Lorenz she her > The previous log was data changed

bruno.corucho · Answer

```response body kind Pod apiVersion v1 metadata name dask root c92b93f5 bw7s8p generateName dask root c92b93f5 b12345 namespace default selfLink api v1 namespaces default pods dask root c34hs723 wd231 uid 0f48a253 a694 4054 bf49 79fc0abcde resourceVersion 7332123 creationTimestamp 2020 07 09T10 23 00Z deletionTimestamp 2020 07 09T10 25 50Z deletionGracePeriodSeconds 30 labels app dask dask root c92b93f5 b worker 9c46ccb0 577b 4658 93fc ec753cs75 a666ba1a 60d7 4c4f 8a73 de0efba50edd user root annotations eks privileged spec volumes name default token jmnsl secret secretName default token jmnsl defaultMode 420 containers name dask worker image args dask worker no bokeh death timeout 60 name 0 env name PREFECT CLOUD GRAPHQL value name PREFECT CLOUD AUTH TOKEN value bG2A7CEA123456789 name PREFECT CONTEXT FLOW RUN ID value 9c46ccb0 577b 4658 93fc 32cdf652xcd name PREFECT CLOUD USE LOCAL SECRETS value false name PREFECT ENGINE FLOW RUNNER DEFAULT CLASS value prefect engine cloud CloudFlowRunner name PREFECT ENGINE TASK RUNNER DEFAULT CLASS value prefect engine cloud CloudTaskRunner name PREFECT ENGINE EXECUTOR DEFAULT CLASS value prefect engine executors DaskExecutor name PREFECT LOGGING LOG TO CLOUD value true name PREFECT LOGGING LEVEL value DEBUG name PREFECT DEBUG value true name DASK DISTRIBUTED SCHEDULER BLOCKED HANDLERS value feed run function name PREFECT LOGGING EXTRA LOGGERS value dask kubernetes core distributed deploy adaptive kubernetes dask kubernetes core distributed deploy adaptive kubernetes name DASK SCHEDULER ADDRESS value resources limits cpu 500m requests cpu 500m volumeMounts name default token jmnsl readOnly true mountPath var run secrets kubernetes io serviceaccount terminationMessagePath dev termination log terminationMessagePolicy File imagePullPolicy IfNotPresent restartPolicy Never terminationGracePeriodSeconds 30 dnsPolicy ClusterFirst serviceAccountName default serviceAccount default nodeName ip 10 12 34 56 78 eu west 1 compute internal securityContext affinity nodeAffinity preferredDuringSchedulingIgnoredDuringExecution weight 100 preference matchExpressions key operator In values worker schedulerName default scheduler tolerations key operator Equal value worker effect NoSchedule key k8s dask org dedicated operator Equal value worker effect NoSchedule key operator Exists effect NoExecute tolerationSeconds 300 key operator Exists effect NoExecute tolerationSeconds 300 priority 0 enableServiceLinks true status phase Running conditions type Initialized status True lastProbeTime null lastTransitionTime 2020 07 09T10 23 00Z type Ready status True lastProbeTime null lastTransitionTime 2020 07 09T10 23 02Z type ContainersReady status True lastProbeTime null lastTransitionTime 2020 07 09T10 23 02Z type PodScheduled status True lastProbeTime null lastTransitionTime 2020 07 09T10 23 00Z hostIP 10 12 34 56 78 podIP 10 12 34 56 78 podIPs ip 10 12 34 56 78 startTime 2020 07 09T10 23 00Z containerStatuses name dask worker state running startedAt 2020 07 09T10 23 01Z lastState ready true restartCount 0 image imageID containerID started true qosClass Burstable ```

bruno.corucho · Answer

```Deleted pod dask root c92b93f5 bw7s8p pod above ``` then ```response body kind Status apiVersion v1 metadata status Failure message services is forbidden User system serviceaccount default default cannot list resource services in API group in the namespace default reason Forbidden details kind services code 403 ```

bruno.corucho · Answer

We re using Dask Kubernetes Environment very much alike the orchestation presentation you did but on aws

bruno.corucho · Answer

If you mean flow info the last messages I got were ```Task store stream 12 finished task run for task with final state Success Flow run SUCCESS all reference tasks succeeded```

bruno.corucho · Answer

Idk if this answers your requests

Laura Lorenz (she/her) · Answer

Ok gotcha My guess is that something during or after pod cleanup on EKS requires this list services permission which it thinks it doesn t have Let me double check on what our standard RBAC is from `prefect agent install kubernetes`

Laura Lorenz (she/her) · Answer

Ok my guess is that you should add the permission to list services in the RBAC for your agent I m guessing you used the default one from <https docs prefect io orchestration agents kubernetes html requirements> so you would need to edit this or otherwise create a new file with a role and rolebinding with this permission and apply it to your kubernetes cluster s default namespace We test mostly on GCP so my suspicion is that EKS permission requirements are slightly different

bruno.corucho · Answer

< Laura Lorenz she her > aaand you were right I updated it and I m no longer having such a problem Thank you Laura

Laura Lorenz (she/her) · Answer

Awesome slightly smiling face