Hello everyone I have been struggling with `DaskKubernetesEn

Question

Hello everyone I have been struggling with `DaskKubernetesEnvironment` and setting imagePullSecrets to pull images from our private gitlab container registry Note I am using a Kubernetes Agent polling from prefect cloud and using prefect version 0 11 2 A couple of questions 1 I thought if I created the docker registry secret using kubernetes added imagePullSecrets to the podSpec and specified a custom scheduler file and worker spec file to DaskKubernetesEnvironment this approach doesn t rely on prefect secrets and I thought should work fine but I am getting an empty dict for imagePullSecrets when inspecting the job and pod specs I thought the kubernetes agent might be overwriting them I saw the `replace job spec yaml` method and so I also specified the secret name in the kubernetes agent manifest under the environment variable IMAGE PULL SECRETS but still I am getting an empty dict for imagePullSecrets any idea why 2 The other approach seems to be not to specify the worker spec and scheduler file but to set private registry to True and docker secret to the name of the Prefect Secret then I use the client to create the prefect secret by setting a name and value I don t see where in the code is the prefect secret s value taken and a kubernetes secret is created so I tried both using a dictionary of docker server docker username docker password and docker email or just setting the value to the name of the k8s secret that I created both approaches didnt work and I am getting an empty dict for imagePullSecrets Any idea what might be going on here and what is the best practice to setting k8s imagePullSecrets for `DaskKubernetesEnvironment`

josh · Answer

Hey < Marwan Sarieddine> I actually think the private registry option on that environment may be outdated needs to be deprecated thinking face will look into that I feel like the environment should instead take the name of an image pull secret instead of trying to create one based on the contents of a Prefect secret On your first point when you say you get the empty dict for imagePullSecrets I m not sure where that would be overwritten as it uses the yaml you provide The image pull secrets option on the Kubernetes agent is for the first `prefect job` that is created and isn t propagated down to the dask scheduler workers Let s try to troubleshoot first question Are you seeing the initial `prefect job` being created and then the following dask scheduler job pod

Marwan Sarieddine · Answer

Hi < josh> thank you for getting back to me so promptly Noted about the private registry option Yes I am seeing a `prefect job` being created to be specific here is how my k8s resources look ```$ kubectl get pods jobs NAME READY STATUS RESTARTS AGE pod prefect agent 5f6458886d z4btq 2 2 Running 0 35m pod prefect job 995c4982 5jzzf 0 1 ErrImagePull 0 6s NAME COMPLETIONS DURATION AGE job batch prefect job 995c4982 0 1 6s 6s```

josh · Answer

If you both add the name of your image pull secret to the Kubernetes agent under `IMAGE PULL SECRETS` and keep the imagePullSecret in the yaml of your environment s scheduler and worker does it start working

Marwan Sarieddine · Answer

that s what I am currently doing but it doesnt seem to be working ```$ kubectl get pod prefect job 995c4982 5jzzf o yaml | yq r spec imagePullSecrets ```

josh · Answer

Can I see the part of your agent deployment yaml that has the `IMAGE PULL SECRETS` set

Marwan Sarieddine · Answer

sure

Marwan Sarieddine · Answer

``` containers args prefect agent start kubernetes command bin bash c env name PREFECT CLOUD API value <https api prefect io> name NAMESPACE value default name IMAGE PULL SECRETS value gitlab secret name PREFECT CLOUD AGENT LABELS value name JOB MEM REQUEST value 256Mi name JOB MEM LIMIT value 512Mi name JOB CPU REQUEST value 500m name JOB CPU LIMIT value 1000m name PREFECT BACKEND value cloud name PREFECT CLOUD AGENT AGENT ADDRESS value http 8080 image prefecthq prefect 0 11 2 python3 6 imagePullPolicy Always livenessProbe failureThreshold 2 httpGet path api health port 8080 initialDelaySeconds 40 periodSeconds 40 name agent resources limits cpu 100m memory 128Mi```

Marwan Sarieddine · Answer

took out the auth token and other stuff that didn t seem relevant

josh · Answer

And could you also show me the output from doing a `kubectl describe` on the prefect job Also removing any auth info of course

Marwan Sarieddine · Answer

```$ kubectl describe pod prefect job 995c4982 5jzzf Name prefect job 995c4982 5jzzf Namespace default Priority 0 Node ip 192 168 79 244 us west 2 compute internal 192 168 79 244 Start Time Wed 20 May 2020 17 55 39 0400 Labels app=prefect job 995c4982 controller uid=5af44c8e 3301 44d3 9890 ffae403ad426 flow run id=868e1b00 8fd5 4394 a3d2 5a2412e2a373 identifier=995c4982 job name=prefect job 995c4982 Annotations eks privileged Status Pending IP 192 168 80 39 IPs lt none gt Controlled By Job prefect job 995c4982 Containers flow Container ID Image Image ID Port lt none gt Host Port lt none gt Command bin sh c Args prefect execute cloud flow State Waiting Reason ImagePullBackOff Ready False Restart Count 0 Limits cpu 1 memory 512Mi Requests cpu 500m memory 256Mi Environment PREFECT CLOUD API PREFECT CONTEXT FLOW RUN ID 868e1b00 8fd5 4394 a3d2 5a2412e2a373 PREFECT CONTEXT FLOW ID 156ce5ef 53c3 4f61 9dcc 004cc890e141 PREFECT CONTEXT NAMESPACE default PREFECT CLOUD AGENT LABELS PREFECT LOGGING LOG TO CLOUD true PREFECT CLOUD USE LOCAL SECRETS false PREFECT LOGGING LEVEL DEBUG PREFECT ENGINE FLOW RUNNER DEFAULT CLASS prefect engine cloud CloudFlowRunner PREFECT ENGINE TASK RUNNER DEFAULT CLASS prefect engine cloud CloudTaskRunner Mounts var run secrets kubernetes io serviceaccount from default token s27ks ro Conditions Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes default token s27ks Type Secret a volume populated by a Secret SecretName default token s27ks Optional false QoS Class Burstable Node Selectors lt none gt Tolerations for 300s for 300s Events Type Reason Age From Message Normal Scheduled 11m default scheduler Successfully assigned default prefect job 995c4982 5jzzf to ip 192 168 79 244 us west 2 compute internal Normal Pulling 9m25s x4 over 10m kubelet ip 192 168 79 244 us west 2 compute internal Pulling image Warning Failed 9m24s x4 over 10m kubelet ip 192 168 79 244 us west 2 compute internal Failed to pull image denied access forbidden Warning Failed 9m24s x4 over 10m kubelet ip 192 168 79 244 us west 2 compute internal Error ErrImagePull Warning Failed 8m57s x7 over 10m kubelet ip 192 168 79 244 us west 2 compute internal Error ImagePullBackOff Normal BackOff 54s x40 over 10m kubelet ip 192 168 79 244 us west 2 compute internal Back off pulling image ```

josh · Answer

Could I also see the output of ```kubectl describe job prefect job 995c4982```

Marwan Sarieddine · Answer

ah sorry ```$ kubectl describe job prefect job 995c4982 Name prefect job 995c4982 Namespace default Selector controller uid=5af44c8e 3301 44d3 9890 ffae403ad426 Labels app=prefect job 995c4982 flow id=156ce5ef 53c3 4f61 9dcc 004cc890e141 flow run id=868e1b00 8fd5 4394 a3d2 5a2412e2a373 identifier=995c4982 Annotations lt none gt Parallelism 1 Completions 1 Start Time Wed 20 May 2020 17 55 39 0400 Pods Statuses 1 Running 0 Succeeded 0 Failed Pod Template Labels app=prefect job 995c4982 controller uid=5af44c8e 3301 44d3 9890 ffae403ad426 flow run id=868e1b00 8fd5 4394 a3d2 5a2412e2a373 identifier=995c4982 job name=prefect job 995c4982 Containers flow Image Port lt none gt Host Port lt none gt Command bin sh c Args prefect execute cloud flow Limits cpu 1 memory 512Mi Requests cpu 500m memory 256Mi Environment PREFECT CLOUD API PREFECT CONTEXT FLOW RUN ID 868e1b00 8fd5 4394 a3d2 5a2412e2a373 PREFECT CONTEXT FLOW ID 156ce5ef 53c3 4f61 9dcc 004cc890e141 PREFECT CONTEXT NAMESPACE default PREFECT CLOUD AGENT LABELS PREFECT LOGGING LOG TO CLOUD true PREFECT CLOUD USE LOCAL SECRETS false PREFECT LOGGING LEVEL DEBUG PREFECT ENGINE FLOW RUNNER DEFAULT CLASS prefect engine cloud CloudFlowRunner PREFECT ENGINE TASK RUNNER DEFAULT CLASS prefect engine cloud CloudTaskRunner Mounts lt none gt Volumes lt none gt Events Type Reason Age From Message Normal SuccessfulCreate 15m job controller Created pod prefect job 995c4982 5jzzf```

josh · Answer

All good One more thing could I see the describe of the agent deployment Same thing with no auth

Marwan Sarieddine · Answer

```$ kubectl describe deployments prefect agent Name prefect agent Namespace default CreationTimestamp Wed 20 May 2020 13 13 47 0400 Labels app=prefect agent Annotations 6 Selector app=prefect agent Replicas 1 desired | 1 updated | 1 total | 1 available | 0 unavailable StrategyType RollingUpdate MinReadySeconds 0 RollingUpdateStrategy 25 max unavailable 25 max surge Pod Template Labels app=prefect agent Containers agent Image prefecthq prefect 0 11 2 python3 6 Port lt none gt Host Port lt none gt Command bin bash c Args prefect agent start kubernetes Limits cpu 100m memory 128Mi Liveness http get http 8080 api health delay=40s timeout=1s period=40s success=1 failure=2 Environment PREFECT CLOUD API NAMESPACE default IMAGE PULL SECRETS gitlab secret PREFECT CLOUD AGENT LABELS JOB MEM REQUEST 256Mi JOB MEM LIMIT 512Mi JOB CPU REQUEST 500m JOB CPU LIMIT 1000m PREFECT BACKEND cloud PREFECT CLOUD AGENT AGENT ADDRESS http 8080 Mounts lt none gt resource manager Image prefecthq prefect 0 11 2 python3 6 Port lt none gt Host Port lt none gt Command bin bash c Args python c from prefect agent kubernetes import ResourceManager ResourceManager start Limits cpu 100m memory 128Mi Environment PREFECT CLOUD API PREFECT CLOUD AGENT RESOURCE MANAGER LOOP INTERVAL 60 NAMESPACE default Mounts lt none gt Volumes lt none gt Conditions Type Status Reason Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets lt none gt NewReplicaSet prefect agent 67b846cc76 1 1 replicas created Events Type Reason Age From Message Normal ScalingReplicaSet 53m deployment controller Scaled up replica set prefect agent 5f6458886d to 1 Normal ScalingReplicaSet 53m deployment controller Scaled down replica set prefect agent 587c5d8cdd to 0 Normal ScalingReplicaSet 11m deployment controller Scaled up replica set prefect agent d6668bb6d to 1 Normal ScalingReplicaSet 11m deployment controller Scaled down replica set prefect agent 5f6458886d to 0 Normal ScalingReplicaSet 7m57s deployment controller Scaled up replica set prefect agent 67b846cc76 to 1 Normal ScalingReplicaSet 7m52s deployment controller Scaled down replica set prefect agent d6668bb6d to 0```

josh · Answer

And just to be certain the `gitlab secret` exists correct smile

Marwan Sarieddine · Answer

lol yep ```$ kubectl get secret o wide NAME TYPE DATA AGE default token s27ks 3 8h gitlab secret 1 5h7m```

josh · Answer

Weird Not sure how but it looks like it isn t attaching that image pull secret The thing that makes it weird is that the code is pretty simple haha and only ``` Use image pull secrets if provided job spec template spec imagePullSecrets 0 name = os getenv IMAGE PULL SECRETS ``` Which it looks like your agent has that env var set My last test would be to check if you can create a pod that uses your image pull secret with something like ```apiVersion v1 kind Pod metadata name private reg spec containers name private reg container image lt your private image gt imagePullSecrets name your secret``` If that pull works then there is a bug in the agent code

Marwan Sarieddine · Answer

ok will check that one other thing I want to check is ``` name IMAGE PULL SECRETS value gitlab secret ``` instead of ``` name IMAGE PULL SECRETS value gitlab secret``` just to be sure it s not because of quotes missing

josh · Answer

Awesome if that is able to pull then could you open an issue with this bug

Marwan Sarieddine · Answer

sure will do thanks for helping in debugging

Marwan Sarieddine · Answer

< josh> not sure what it was but it is working now man shrugging it is not the quotes at least I dont think so because it says ```deployment apps prefect agent unchanged``` when I add the quotes

josh · Answer

Awesome

Marwan Sarieddine · Answer

thanks again for the help glad it is not a bug

josh · Answer

Anytime Haha I m glad it s not a bug as well wink