bruno.corucho
07/09/2020, 10:35 AMresponse body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"services\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"kind":"services"},"code":403}
PS: We can see "sucessful" response bodies, but should we ignore this last entry / is it intended?
Thanks in advance.Laura Lorenz (she/her)
07/09/2020, 1:40 PMbruno.corucho
07/09/2020, 2:00 PMresponse body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"dask-root-c92b93f5-bw7s8p","generateName":"dask-root-c92b93f5-b12345","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/dask-root-c34hs723-wd231","uid":"0f48a253-a694-4054-bf49-79fc0abcde","resourceVersion":"7332123","creationTimestamp":"2020-07-09T10:23:00Z","deletionTimestamp":"2020-07-09T10:25:50Z","deletionGracePeriodSeconds":30,"labels":{"app":"dask","<http://dask.org/cluster-name|dask.org/cluster-name>":"dask-root-c92b93f5-b","<http://dask.org/component|dask.org/component>":"worker","<http://prefect.io/flow_run_id|prefect.io/flow_run_id>":"9c46ccb0-577b-4658-93fc-ec753cs75","<http://prefect.io/identifier|prefect.io/identifier>":"a666ba1a-60d7-4c4f-8a73-de0efba50edd","user":"root"},"annotations":{"<http://kubernetes.io/psp|kubernetes.io/psp>":"eks.privileged"}},"spec":{"volumes":[{"name":"default-token-jmnsl","secret":{"secretName":"default-token-jmnsl","defaultMode":420}}],"containers":[{"name":"dask-worker","image":"<http://65999014212345.dkr.ecr.eu-west-1.amazonaws.com/strdata-flow:2020-07-09t10-21-22-745632-00-00|65999014212345.dkr.ecr.eu-west-1.amazonaws.com/strdata-flow:2020-07-09t10-21-22-745632-00-00>","args":["dask-worker","--no-bokeh","--death-timeout","60","--name","0"],"env":[{"name":"PREFECT__CLOUD__GRAPHQL","value":"<https://api.prefect.io/graphql>"},{"name":"PREFECT__CLOUD__AUTH_TOKEN","value":"bG2A7CEA123456789"},{"name":"PREFECT__CONTEXT__FLOW_RUN_ID","value":"9c46ccb0-577b-4658-93fc-32cdf652xcd"},{"name":"PREFECT__CLOUD__USE_LOCAL_SECRETS","value":"false"},{"name":"PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS","value":"prefect.engine.cloud.CloudFlowRunner"},{"name":"PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS","value":"prefect.engine.cloud.CloudTaskRunner"},{"name":"PREFECT__ENGINE__EXECUTOR__DEFAULT_CLASS","value":"prefect.engine.executors.DaskExecutor"},{"name":"PREFECT__LOGGING__LOG_TO_CLOUD","value":"true"},{"name":"PREFECT__LOGGING__LEVEL","value":"DEBUG"},{"name":"PREFECT__DEBUG","value":"true"},{"name":"DASK_DISTRIBUTED__SCHEDULER__BLOCKED_HANDLERS","value":"['feed', 'run_function']"},{"name":"PREFECT__LOGGING__EXTRA_LOGGERS","value":"['dask_kubernetes.core', 'distributed.deploy.adaptive', 'kubernetes', 'dask_kubernetes.core', 'distributed.deploy.adaptive', 'kubernetes']"},{"name":"DASK_SCHEDULER_ADDRESS","value":"<tcp://10.12.34.56.78:12345>"}],"resources":{"limits":{"cpu":"500m"},"requests":{"cpu":"500m"}},"volumeMounts":[{"name":"default-token-jmnsl","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"ip-10.12.34.56.78.eu-west-1.compute.internal","securityContext":{},"affinity":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"preference":{"matchExpressions":[{"key":"<http://k8s.dask.org/node-purpose|k8s.dask.org/node-purpose>","operator":"In","values":["worker"]}]}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"<http://k8s.dask.org/dedicated|k8s.dask.org/dedicated>","operator":"Equal","value":"worker","effect":"NoSchedule"},{"key":"k8s.dask.org_dedicated","operator":"Equal","value":"worker","effect":"NoSchedule"},{"key":"<http://node.kubernetes.io/not-ready|node.kubernetes.io/not-ready>","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"<http://node.kubernetes.io/unreachable|node.kubernetes.io/unreachable>","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-07-09T10:23:00Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-07-09T10:23:02Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-07-09T10:23:02Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-07-09T10:23:00Z"}],"hostIP":"10.12.34.56.78","podIP":"10.12.34.56.78","podIPs":[{"ip":"10.12.34.56.78"}],"startTime":"2020-07-09T10:23:00Z","containerStatuses":[{"name":"dask-worker","state":{"running":{"startedAt":"2020-07-09T10:23:01Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"<http://659990142216.dkr.ecr.eu-west-1.amazonaws.com/strdata-flow:2020-07-09t10-21-22-745632-00-00|659990142216.dkr.ecr.eu-west-1.amazonaws.com/strdata-flow:2020-07-09t10-21-22-745632-00-00>","imageID":"<docker-pullable://123456789.dkr.ecr.eu-west-1.amazonaws.com/strdata-flow@sha256:e1ac7a8fd8acb6173e1c3d9617e0da4494c1d885>","containerID":"<docker://0af5315fda9959bdfb267cb4bbc9b742805da6b4c347479da8579>","started":true}],"qosClass":"Burstable"}}
Deleted pod: dask-root-c92b93f5-bw7s8p (pod above)
then
response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"services\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"kind":"services"},"code":403}
Task 'store_stream[12]': finished task run for task with final state: 'Success'
Flow run SUCCESS: all reference tasks succeeded
Laura Lorenz (she/her)
07/09/2020, 2:23 PMprefect agent install kubernetes
bruno.corucho
07/10/2020, 9:04 AMLaura Lorenz (she/her)
07/10/2020, 11:26 AM