Jafar A
03/02/2023, 8:33 PMprefect deployment build flows/main.py:ingest_everflow \
-n prefect-k8s-example -q k8s \
-sb s3/s3-everflow/ef_testing/docker-k8s-image \
-ib kubernetes-job/prefect-k8s-example \
--apply
14:40:49.825 | INFO | prefect.agent - Submitting flow run '99f81791-abda-488f-802e-d949535cb351'
/Users/jafaramin_1/env/lib/python3.10/site-packages/prefect/agent.py:216: UserWarning: Block document has schema checksum sha256:1a553852c1bcc9cd95645917bdbaaae9e28cbcc1270ff5b43b45bbd87564f5cc which does not match the schema checksum for class 'KubernetesJob'. This indicates the schema has changed and this block may not load.
infrastructure_block = Block._from_block_document(infra_document)
14:40:50.538 | INFO | prefect.agent - Completed submission of flow run '99f81791-abda-488f-802e-d949535cb351'
14:40:50.546 | DEBUG | prefect.infrastructure.kubernetes-job - Job 'prefect-k8s-example-mwmgm': Starting watch for pod start...
14:40:50.555 | INFO | prefect.infrastructure.kubernetes-job - Job 'prefect-k8s-example-mwmgm': Pod has status 'Pending'.
14:40:54.827 | DEBUG | prefect.agent - Checking for flow runs...
14:40:59.925 | DEBUG | prefect.agent - Checking for flow runs...
14:42:50.560 | ERROR | prefect.infrastructure.kubernetes-job - Job 'prefect-k8s-example-mwmgm': Pod never started.
Jeff Hale
03/02/2023, 8:45 PM-i
with extra pip packages environment variable to run K8s on the latest Docker image - does that work?
The only other thing that stands out to me that might be related is that your storage block slug is “s3/s3-everflow/ef_testing/docker-k8s-image” - is that the slug that comes up if you do prefect block ls
?Jafar A
03/02/2023, 9:10 PMprefect deployment build flows/main.py:ingest_everflow -n prefect-k8s-example -q k8s -sb s3/s3-everflow/ef_testing/docker-k8s-image -i kubernetes-job --override env.EXTRA_PIP_PACKAGES=s3fs --apply
yes s3/s3-everflow
is the slug and added some subfolder for s3 and I think it's optional (i wanted a copy of code being sent to s3 folder) . I tried removing sb from deployment and still got "Pod never started" error.Jeff Hale
03/02/2023, 9:17 PMJafar A
03/02/2023, 9:21 PMkubectl get all -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/organic-foxhound-xsl22-2tc2s 0/1 Completed 0 15m
default pod/prefect-k8s-example-8wn2j-rzj72 0/1 ImagePullBackOff 0 10m
default pod/prefect-k8s-example-d5mhb-mcpsx 0/1 ImagePullBackOff 0 107m
default pod/prefect-k8s-example-ecr-9wlmj-7fd9z 0/1 ImagePullBackOff 0 2m34s
default pod/prefect-k8s-example-mwmgm-j5mtx 0/1 ImagePullBackOff 0 97m
Jeff Hale
03/02/2023, 9:23 PMJafar A
03/02/2023, 9:28 PMJeff Hale
03/03/2023, 2:47 PMChristopher Boyd
03/03/2023, 2:51 PMkubectl get events
kubectl describe pod prefect-k8s-example-8wn2j-rzj72
Jafar A
03/03/2023, 6:31 PM(env) MacBook-Pro:de-orchestration jamin$ kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE
3m40s Normal BackOff pod/prefect-k8s-example-8wn2j-rzj72 Back-off pulling image "prefect-docker-example:latest"
5m54s Normal Pulling pod/prefect-k8s-example-d5mhb-mcpsx Pulling image "prefect-docker-example:latest"
61s Normal BackOff pod/prefect-k8s-example-d5mhb-mcpsx Back-off pulling image "prefect-docker-example:latest"
51s Normal BackOff pod/prefect-k8s-example-ecr-9wlmj-7fd9z Back-off pulling image "<http://880954911558.dkr.ecr.us-west-2.amazonaws.com/prefect-docker-example:latest|880954911558.dkr.ecr.us-west-2.amazonaws.com/prefect-docker-example:latest>"
72s Normal BackOff pod/prefect-k8s-example-mwmgm-j5mtx Back-off pulling image "prefect-docker-example:latest"
(env) MacBook-Pro:de-orchestration jamin$
(env) MacBook-Pro:de-orchestration jamin$ kubectl describe pod prefect-k8s-example-8wn2j-rzj72
Name: prefect-k8s-example-8wn2j-rzj72
Namespace: default
Priority: 0
Service Account: default
Node: docker-desktop/192.168.65.4
Start Time: Thu, 02 Mar 2023 16:08:19 -0500
Labels: controller-uid=28d05e99-61f4-4554-b22b-d1ea723bbe00
job-name=prefect-k8s-example-8wn2j
Annotations: <none>
Status: Pending
IP: 10.1.0.9
IPs:
IP: 10.1.0.9
Controlled By: Job/prefect-k8s-example-8wn2j
Containers:
prefect-job:
Container ID:
Image: prefect-docker-example:latest
Image ID:
Port: <none>
Host Port: <none>
Args:
python
-m
prefect.engine
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment:
PREFECT_API_URL: <https://api.prefect.cloud/api/accounts/1ce80bf4-595f-42af-9feb-85431f7631a7/workspaces/2f7cd4bc-77a6-4b3b-8d68-dca0941eaae5>
PREFECT_API_KEY: @@@@@@@@@@@@
PREFECT_LOGGING_LEVEL: DEBUG
PREFECT__FLOW_RUN_ID: 3e6cb334106a40de816ea42b40ba31b7
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2vxvl (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-2vxvl:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal BackOff 3m57s (x1519 over 19h) kubelet Back-off pulling image "prefect-docker-example:latest"
Christopher Boyd
03/03/2023, 6:36 PMprefect-docker-example:latest
Jafar A
03/03/2023, 6:45 PMFROM prefecthq/prefect:2-python3.10
Are you saying this is not a base prefect image?Christopher Boyd
03/03/2023, 6:48 PMJafar A
03/03/2023, 7:17 PMImage
prefect-docker-example:latest
I guess question is now that dockerhub has my image, how do I reference the dockerhub image in my deploymentChristopher Boyd
03/03/2023, 7:29 PMjafarsaedi/prefect-docker-example
jafarsaedi
would be your repository, prefect-docker-example
would be the tag:latest
tagged at the end, a pull will always reference :latestJafar A
03/03/2023, 7:33 PM