https://prefect.io logo
Join Slack
Powered by
Hello, I’m getting the following error with DaskKu...
# ask-community
n
Hello, I’m getting the following error with DaskKubernetes environment. It has to do with the environment not being able to read the secret in my prefect k8s agent:
Copy code
Docker registry secret default-docker does not exist for this tenant.

Creating Docker registry kubernetes secret from "prefect" Prefect Secret.

Failed to load and execute Flow's environment: ClientError([{'message': 'No value found for the requested key', 'locations': [{'line': 2, 'column': 9}], 'path': ['secret_value'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'errors': [{'message': 'No value found for the requested key', 'locations': [], 'path': ['secret_value']}]}}}])
Here is my flow:
Copy code
from prefect import task, Flow, Parameter
from prefect import Client
from prefect.environments.storage import Docker
from prefect.environments import DaskKubernetesEnvironment
import json

@task(log_stdout=True)
def say_hello():
    print("Hello, K8s!".format(name))
    return True

@task
def print_completed(status):
    print('Success')

flow = Flow("Simple-Flow", 
    environment=DaskKubernetesEnvironment(min_workers=1, max_workers=3, private_registry=True, docker_secret='prefect'), 
    storage=Docker(image_name="hello-prefect", registry_url='<http://my-company-reg.com|my-company-reg.com>'))

print_completed.set_upstream(say_hello, flow=flow)
print_completed.bind(status=say_hello, flow=flow)

flow.register(project_name="Hello K8s")
And here is the description of my agent’s pod:
Copy code
Name:               prefect-agent-7fd7ff499-nzvlr
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               **********
Start Time:         Thu, 21 May 2020 15:51:33 -0400
Labels:             app=prefect-agent
                    pod-template-hash=7fd7ff499
Annotations:        <http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged
Status:             Running
IP:                 100.64.16.213
Controlled By:      ReplicaSet/prefect-agent-7fd7ff499
Containers:
  agent:
    Container ID:  <docker://d6a7fd06ea04adbf956d7a4f9aefdaf421929f1bbaf2b6af4e90a9262095fe4>f
    Image:         prefecthq/prefect:0.11.2-python3.6
    Image ID:      <docker-pullable://prefecthq/prefect@sha256:d0f685016f5f82a373a0b3aeadb4598529e7f31139ca9f585b2077e1f6097c64>
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -c
    Args:
      prefect agent start kubernetes
    State:          Running
      Started:      Thu, 21 May 2020 15:51:34 -0400
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  128Mi
    Requests:
      cpu:     100m
      memory:  128Mi
    Liveness:  http-get http://:8080/api/health delay=40s timeout=1s period=40s #success=1 #failure=2
    Environment:
      PREFECT__CLOUD__AGENT__AUTH_TOKEN:     ********
      PREFECT__CLOUD__API:                   <https://api.prefect.io>
      NAMESPACE:                             default
      IMAGE_PULL_SECRETS:                    prefect
      PREFECT__CLOUD__AGENT__LABELS:         []
      JOB_MEM_REQUEST:                       
      JOB_MEM_LIMIT:                         
      JOB_CPU_REQUEST:                       
      JOB_CPU_LIMIT:                         
      PREFECT__BACKEND:                      cloud
      PREFECT__CLOUD__AGENT__AGENT_ADDRESS:  http://:8080
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5nvmj (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-5nvmj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5nvmj
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> for 300s
                 <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> for 300s
Events:          <none>
I’ve verified that the secret named 
prefect
pulls down my image from our company’s registry.
👀 1
d
Hi @Noah Nethery, What version of Prefect are you running? If you’re running off of 
master
, you might be encountering some trouble with this PR: https://github.com/PrefectHQ/prefect/pull/2630/files
I think we’re officially recommending that you should set imagePullSecrets as described here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-pod-that-uses-your-secret
n
@Dylan does this mean I will be using the custom configuration from a yaml in my Dask Kubernetes environment initialization?
By the way, I’m on prefect version 0.11.2 according to pip
@Dylan I also have an addendum. Here is a new error I am getting on Prefect:
Copy code
Unexpected error raised during flow run: 'env'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/environments/execution/dask/k8s.py", line 277, in run_flow
    worker_pod = self._populate_worker_spec_yaml(yaml_obj=worker_pod)
  File "/usr/local/lib/python3.7/site-packages/prefect/environments/execution/dask/k8s.py", line 555, in _populate_worker_spec_yaml
    env = yaml_obj["spec"]["containers"][0]["env"]
KeyError: 'env'
Here is my new flow:
Copy code
from prefect import task, Flow, Parameter
from prefect import Client
from prefect.environments.storage import Docker
from prefect.environments import DaskKubernetesEnvironment

@task(log_stdout=True)
def say_hello():
    print("Hello, K8s!".format(name))
    return True

@task
def print_completed(status):
    print('Success')

flow = Flow("Simple-Flow", 
    environment=DaskKubernetesEnvironment(min_workers=1, max_workers=3, worker_spec_file="worker-spec.yaml"), 
    storage=Docker(image_name="hello-prefect", registry_url='*********'))

print_completed.set_upstream(say_hello, flow=flow)
print_completed.bind(status=say_hello, flow=flow)

flow.register(project_name="Hello K8s")
And here is my worker-spec.yaml:
Copy code
kind: Pod
spec:
  restartPolicy: Never
  containers:
  - image: *********/hello-prefect:latest
    imagePullSecrets:
    - name: prefect
    imagePullPolicy: IfNotPresent
    args: [dask-worker, --nthreads, '2', --no-dashboard, --memory-limit, 6GB, --death-timeout, '60']
    name: dask-test
    resources:
      limits:
        cpu: "2"
        memory: 6G
      requests:
        cpu: "2"
        memory: 6G
z
Hi Noah, correct-- you'll want to specify custom worker and scheduler YAML if you're making use of image pull secrets. For the new error you're getting, it looks like your worker spec is missing an env section. The example YAML below might be helpful:
Copy code
spec:
  restartPolicy: Never
  containers:
  - image: prefecthq/prefect:latest
    imagePullPolicy: IfNotPresent
    args: ...
    name: dask-worker
    env: {}
d
@Noah Nethery Zachary’s got you covered ☝️
n
@Zachary Hughes Thank you, I was able to get this implemented but now I’m back to this error:
Copy code
Failed to load and execute Flow's environment: ClientError([{'message': 'No value found for the requested key', 'locations': [{'line': 2, 'column': 9}], 'path': ['secret_value'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'errors': [{'message': 'No value found for the requested key', 'locations': [], 'path': ['secret_value']}]}}}])
I still don’t understand since my k8s agent has my secret, my Docker storage line in my flow has the secret name now, and the worker-spec.yaml has the secret which I’ve confirmed works since it’s the pod from the job is able to pull the image no problem:
Copy code
Name:               prefect-job-456a6ef7-7qdds
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               *********
Start Time:         Tue, 26 May 2020 15:54:17 -0400
Labels:             app=prefect-job-456a6ef7
                    controller-uid=af02435b-9f8a-11ea-998a-12e15268c2e9
                    flow_run_id=bdc7b4e5-d6f1-424a-b74c-0b2d0e480720
                    identifier=456a6ef7
                    job-name=prefect-job-456a6ef7
Annotations:        <http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged
Status:             Failed
IP:                 100.64.208.103
Controlled By:      Job/prefect-job-456a6ef7
Containers:
  flow:
    Container ID:  <docker://1890b2a69ac2861349343e8bb0c7d3138639fa509139a0cd73c083d53196c48>7
    Image:         <http://itx-bke.artifactrepo.jnj.com/hello-prefect:latest|itx-bke.artifactrepo.jnj.com/hello-prefect:latest>
    Image ID:      docker-pullable://*********/hello-prefect@sha256:7f32a4dbf900b87d85b9b29f778868edfea2d2bfe733e9b15b04feb7c1a29393
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
    Args:
      prefect execute cloud-flow
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 26 May 2020 15:54:22 -0400
      Finished:     Tue, 26 May 2020 15:54:27 -0400
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:  100m
    Requests:
      cpu:  100m
    Environment:
      PREFECT__CLOUD__API:                          <https://api.prefect.io>
      PREFECT__CLOUD__AUTH_TOKEN:                   ********
      PREFECT__CONTEXT__FLOW_RUN_ID:                bdc7b4e5-d6f1-424a-b74c-0b2d0e480720
      PREFECT__CONTEXT__FLOW_ID:                    2f3c87ce-851d-4088-9af9-8051274d61d3
      PREFECT__CONTEXT__NAMESPACE:                  default
      PREFECT__CLOUD__AGENT__LABELS:                []
      PREFECT__LOGGING__LOG_TO_CLOUD:               true
      PREFECT__CLOUD__USE_LOCAL_SECRETS:            false
      PREFECT__LOGGING__LEVEL:                      DEBUG
      PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS:  prefect.engine.cloud.CloudFlowRunner
      PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS:  prefect.engine.cloud.CloudTaskRunner
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5nvmj (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-5nvmj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5nvmj
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> for 300s
                 <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> for 300s
Events:
  Type    Reason     Age    From                                     Message
  ----    ------     ----   ----                                     -------
  Normal  Scheduled  3m26s  default-scheduler                        Successfully assigned default/prefect-job-456a6ef7-7qdds to ip-100-64-223-131.ec2.internal
  Normal  Pulling    3m25s  kubelet, ip-100-64-223-131.ec2.internal  Pulling image "******/hello-prefect:latest"
  Normal  Pulled     3m21s  kubelet, ip-100-64-223-131.ec2.internal  Successfully pulled image "******/hello-prefect:latest"
  Normal  Created    3m21s  kubelet, ip-100-64-223-131.ec2.internal  Created container flow
  Normal  Started    3m21s  kubelet, ip-100-64-223-131.ec2.internal  Started container flow
z
Hmm. The error you're posting looks like it's coming from the Cloud API. Is there anywhere in your flow you've set a dependency on a secret stored in Cloud?
n
@Zachary Hughes is there a way for me say to use local secrets instead of cloud ones?
z
@Noah Nethery This PR should help you out as soon as it's merged. In the meantime, you could specify the secret via an environment variable or create a Cloud secret, but I recognize that neither of those are particularly satisfying solutions. https://github.com/PrefectHQ/prefect/pull/2657
🤩 1
Open in Slack
PreviousNext
Powered by