Noah Nethery
05/21/2020, 9:27 PMDocker registry secret default-docker does not exist for this tenant.
Creating Docker registry kubernetes secret from "prefect" Prefect Secret.
Failed to load and execute Flow's environment: ClientError([{'message': 'No value found for the requested key', 'locations': [{'line': 2, 'column': 9}], 'path': ['secret_value'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'errors': [{'message': 'No value found for the requested key', 'locations': [], 'path': ['secret_value']}]}}}])
Here is my flow:
from prefect import task, Flow, Parameter
from prefect import Client
from prefect.environments.storage import Docker
from prefect.environments import DaskKubernetesEnvironment
import json
@task(log_stdout=True)
def say_hello():
print("Hello, K8s!".format(name))
return True
@task
def print_completed(status):
print('Success')
flow = Flow("Simple-Flow",
environment=DaskKubernetesEnvironment(min_workers=1, max_workers=3, private_registry=True, docker_secret='prefect'),
storage=Docker(image_name="hello-prefect", registry_url='<http://my-company-reg.com|my-company-reg.com>'))
print_completed.set_upstream(say_hello, flow=flow)
print_completed.bind(status=say_hello, flow=flow)
flow.register(project_name="Hello K8s")
And here is the description of my agent’s pod:
Name: prefect-agent-7fd7ff499-nzvlr
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: **********
Start Time: Thu, 21 May 2020 15:51:33 -0400
Labels: app=prefect-agent
pod-template-hash=7fd7ff499
Annotations: <http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged
Status: Running
IP: 100.64.16.213
Controlled By: ReplicaSet/prefect-agent-7fd7ff499
Containers:
agent:
Container ID: <docker://d6a7fd06ea04adbf956d7a4f9aefdaf421929f1bbaf2b6af4e90a9262095fe4>f
Image: prefecthq/prefect:0.11.2-python3.6
Image ID: <docker-pullable://prefecthq/prefect@sha256:d0f685016f5f82a373a0b3aeadb4598529e7f31139ca9f585b2077e1f6097c64>
Port: <none>
Host Port: <none>
Command:
/bin/bash
-c
Args:
prefect agent start kubernetes
State: Running
Started: Thu, 21 May 2020 15:51:34 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 100m
memory: 128Mi
Requests:
cpu: 100m
memory: 128Mi
Liveness: http-get http://:8080/api/health delay=40s timeout=1s period=40s #success=1 #failure=2
Environment:
PREFECT__CLOUD__AGENT__AUTH_TOKEN: ********
PREFECT__CLOUD__API: <https://api.prefect.io>
NAMESPACE: default
IMAGE_PULL_SECRETS: prefect
PREFECT__CLOUD__AGENT__LABELS: []
JOB_MEM_REQUEST:
JOB_MEM_LIMIT:
JOB_CPU_REQUEST:
JOB_CPU_LIMIT:
PREFECT__BACKEND: cloud
PREFECT__CLOUD__AGENT__AGENT_ADDRESS: http://:8080
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5nvmj (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-5nvmj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5nvmj
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> for 300s
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> for 300s
Events: <none>
I’ve verified that the secret named prefect
pulls down my image from our company’s registry.Dylan
master
, you might be encountering some trouble with this PR: https://github.com/PrefectHQ/prefect/pull/2630/filesNoah Nethery
05/26/2020, 5:38 PMUnexpected error raised during flow run: 'env'
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/environments/execution/dask/k8s.py", line 277, in run_flow
worker_pod = self._populate_worker_spec_yaml(yaml_obj=worker_pod)
File "/usr/local/lib/python3.7/site-packages/prefect/environments/execution/dask/k8s.py", line 555, in _populate_worker_spec_yaml
env = yaml_obj["spec"]["containers"][0]["env"]
KeyError: 'env'
Here is my new flow:
from prefect import task, Flow, Parameter
from prefect import Client
from prefect.environments.storage import Docker
from prefect.environments import DaskKubernetesEnvironment
@task(log_stdout=True)
def say_hello():
print("Hello, K8s!".format(name))
return True
@task
def print_completed(status):
print('Success')
flow = Flow("Simple-Flow",
environment=DaskKubernetesEnvironment(min_workers=1, max_workers=3, worker_spec_file="worker-spec.yaml"),
storage=Docker(image_name="hello-prefect", registry_url='*********'))
print_completed.set_upstream(say_hello, flow=flow)
print_completed.bind(status=say_hello, flow=flow)
flow.register(project_name="Hello K8s")
And here is my worker-spec.yaml:
kind: Pod
spec:
restartPolicy: Never
containers:
- image: *********/hello-prefect:latest
imagePullSecrets:
- name: prefect
imagePullPolicy: IfNotPresent
args: [dask-worker, --nthreads, '2', --no-dashboard, --memory-limit, 6GB, --death-timeout, '60']
name: dask-test
resources:
limits:
cpu: "2"
memory: 6G
requests:
cpu: "2"
memory: 6G
Zachary Hughes
05/26/2020, 7:20 PMspec:
restartPolicy: Never
containers:
- image: prefecthq/prefect:latest
imagePullPolicy: IfNotPresent
args: ...
name: dask-worker
env: {}
Dylan
Noah Nethery
05/26/2020, 7:59 PMFailed to load and execute Flow's environment: ClientError([{'message': 'No value found for the requested key', 'locations': [{'line': 2, 'column': 9}], 'path': ['secret_value'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'errors': [{'message': 'No value found for the requested key', 'locations': [], 'path': ['secret_value']}]}}}])
I still don’t understand since my k8s agent has my secret, my Docker storage line in my flow has the secret name now, and the worker-spec.yaml has the secret which I’ve confirmed works since it’s the pod from the job is able to pull the image no problem:Name: prefect-job-456a6ef7-7qdds
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: *********
Start Time: Tue, 26 May 2020 15:54:17 -0400
Labels: app=prefect-job-456a6ef7
controller-uid=af02435b-9f8a-11ea-998a-12e15268c2e9
flow_run_id=bdc7b4e5-d6f1-424a-b74c-0b2d0e480720
identifier=456a6ef7
job-name=prefect-job-456a6ef7
Annotations: <http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged
Status: Failed
IP: 100.64.208.103
Controlled By: Job/prefect-job-456a6ef7
Containers:
flow:
Container ID: <docker://1890b2a69ac2861349343e8bb0c7d3138639fa509139a0cd73c083d53196c48>7
Image: <http://itx-bke.artifactrepo.jnj.com/hello-prefect:latest|itx-bke.artifactrepo.jnj.com/hello-prefect:latest>
Image ID: docker-pullable://*********/hello-prefect@sha256:7f32a4dbf900b87d85b9b29f778868edfea2d2bfe733e9b15b04feb7c1a29393
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
Args:
prefect execute cloud-flow
State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 26 May 2020 15:54:22 -0400
Finished: Tue, 26 May 2020 15:54:27 -0400
Ready: False
Restart Count: 0
Limits:
cpu: 100m
Requests:
cpu: 100m
Environment:
PREFECT__CLOUD__API: <https://api.prefect.io>
PREFECT__CLOUD__AUTH_TOKEN: ********
PREFECT__CONTEXT__FLOW_RUN_ID: bdc7b4e5-d6f1-424a-b74c-0b2d0e480720
PREFECT__CONTEXT__FLOW_ID: 2f3c87ce-851d-4088-9af9-8051274d61d3
PREFECT__CONTEXT__NAMESPACE: default
PREFECT__CLOUD__AGENT__LABELS: []
PREFECT__LOGGING__LOG_TO_CLOUD: true
PREFECT__CLOUD__USE_LOCAL_SECRETS: false
PREFECT__LOGGING__LEVEL: DEBUG
PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS: prefect.engine.cloud.CloudFlowRunner
PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS: prefect.engine.cloud.CloudTaskRunner
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5nvmj (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-5nvmj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5nvmj
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> for 300s
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m26s default-scheduler Successfully assigned default/prefect-job-456a6ef7-7qdds to ip-100-64-223-131.ec2.internal
Normal Pulling 3m25s kubelet, ip-100-64-223-131.ec2.internal Pulling image "******/hello-prefect:latest"
Normal Pulled 3m21s kubelet, ip-100-64-223-131.ec2.internal Successfully pulled image "******/hello-prefect:latest"
Normal Created 3m21s kubelet, ip-100-64-223-131.ec2.internal Created container flow
Normal Started 3m21s kubelet, ip-100-64-223-131.ec2.internal Started container flow
Zachary Hughes
05/26/2020, 8:07 PMNoah Nethery
05/26/2020, 8:30 PMZachary Hughes
05/26/2020, 8:49 PM