https://prefect.io logo
Join Slack
Powered by
Hello, I've had good success deploying with prefec...
# best-practices
j
Hello, I've had good success deploying with prefect in subprocess, docker locally and also with ECR. Now when it comes to k8s, docker desktop k8s is running (green with k8s_POD_prefect-k8s-... running) but prefect says Pod never started! I followed a post by @Jeff Hale and this is my build: 
prefect deployment build flows/main.py:ingest_everflow \
-n prefect-k8s-example -q k8s \
-sb s3/s3-everflow/ef_testing/docker-k8s-image \
-ib kubernetes-job/prefect-k8s-example \
--apply
Copy code
14:40:49.825 | INFO    | prefect.agent - Submitting flow run '99f81791-abda-488f-802e-d949535cb351'
/Users/jafaramin_1/env/lib/python3.10/site-packages/prefect/agent.py:216: UserWarning: Block document has schema checksum sha256:1a553852c1bcc9cd95645917bdbaaae9e28cbcc1270ff5b43b45bbd87564f5cc which does not match the schema checksum for class 'KubernetesJob'. This indicates the schema has changed and this block may not load.
  infrastructure_block = Block._from_block_document(infra_document)
14:40:50.538 | INFO    | prefect.agent - Completed submission of flow run '99f81791-abda-488f-802e-d949535cb351'
14:40:50.546 | DEBUG   | prefect.infrastructure.kubernetes-job - Job 'prefect-k8s-example-mwmgm': Starting watch for pod start...
14:40:50.555 | INFO    | prefect.infrastructure.kubernetes-job - Job 'prefect-k8s-example-mwmgm': Pod has status 'Pending'.
14:40:54.827 | DEBUG   | prefect.agent - Checking for flow runs...
14:40:59.925 | DEBUG   | prefect.agent - Checking for flow runs...
14:42:50.560 | ERROR   | prefect.infrastructure.kubernetes-job - Job 'prefect-k8s-example-mwmgm': Pod never started.
j
The exact same deployment (except for changing the infrastructure block) works okay with Docker locally, but not K8s locally? What if you use 
-i
with extra pip packages environment variable to run K8s on the latest Docker image - does that work? The only other thing that stands out to me that might be related is that your storage block slug is “s3/s3-everflow/ef_testing/docker-k8s-image” - is that the slug that comes up if you do 
prefect block ls
?
j
@Jeff Hale yes same image of packages (local or pulled from ecr) runs ok with docker block but not working with k8s block. trying adding -i and quite frankly this is my first k8s experience so not much previous knowledge and I'm getting error asking to install other packages (pandas,..ect) 
prefect deployment build flows/main.py:ingest_everflow   -n prefect-k8s-example   -q k8s   -sb s3/s3-everflow/ef_testing/docker-k8s-image   -i kubernetes-job --override env.EXTRA_PIP_PACKAGES=s3fs  --apply
yes 
s3/s3-everflow
is the slug and added some subfolder for s3 and I think it's optional (i wanted a copy of code being sent to s3 folder) . I tried removing sb from deployment and still got "Pod never started" error.
j
Hmm. If same thing worked with Docker, it is probably an issue with your k8s running locally. Do kubectl commands work?
j
kubectl still works 
kubectl get all -A
NAMESPACE     NAME                                         READY   STATUS             RESTARTS      AGE
default       pod/organic-foxhound-xsl22-2tc2s             0/1     Completed          0             15m
default       pod/prefect-k8s-example-8wn2j-rzj72          0/1     ImagePullBackOff   0             10m
default       pod/prefect-k8s-example-d5mhb-mcpsx          0/1     ImagePullBackOff   0             107m
default       pod/prefect-k8s-example-ecr-9wlmj-7fd9z      0/1     ImagePullBackOff   0             2m34s
default       pod/prefect-k8s-example-mwmgm-j5mtx          0/1     ImagePullBackOff   0             97m
j
Are there any logs in the containers in Docker Desktop?
j
docker logs containerid returns logs for successful docker-deployment . when I try it on k8s container, return none
j
hmm. @Christopher Boyd is a resident K8s expert. Any thoughts here?
c
What do the following say:
Copy code
kubectl get events
kubectl describe pod prefect-k8s-example-8wn2j-rzj72
ImagePullBackOff says it can’t pull the image, describe / events should say why
what image are you using
j
@Christopher Boyd you're right and here is the output however description is not clear
Copy code
(env) MacBook-Pro:de-orchestration jamin$ kubectl get events
LAST SEEN   TYPE     REASON    OBJECT                                    MESSAGE
3m40s       Normal   BackOff   pod/prefect-k8s-example-8wn2j-rzj72       Back-off pulling image "prefect-docker-example:latest"
5m54s       Normal   Pulling   pod/prefect-k8s-example-d5mhb-mcpsx       Pulling image "prefect-docker-example:latest"
61s         Normal   BackOff   pod/prefect-k8s-example-d5mhb-mcpsx       Back-off pulling image "prefect-docker-example:latest"
51s         Normal   BackOff   pod/prefect-k8s-example-ecr-9wlmj-7fd9z   Back-off pulling image "<http://880954911558.dkr.ecr.us-west-2.amazonaws.com/prefect-docker-example:latest|880954911558.dkr.ecr.us-west-2.amazonaws.com/prefect-docker-example:latest>"
72s         Normal   BackOff   pod/prefect-k8s-example-mwmgm-j5mtx       Back-off pulling image "prefect-docker-example:latest"
(env) MacBook-Pro:de-orchestration jamin$ 
(env) MacBook-Pro:de-orchestration jamin$ kubectl describe pod prefect-k8s-example-8wn2j-rzj72
Name:             prefect-k8s-example-8wn2j-rzj72
Namespace:        default
Priority:         0
Service Account:  default
Node:             docker-desktop/192.168.65.4
Start Time:       Thu, 02 Mar 2023 16:08:19 -0500
Labels:           controller-uid=28d05e99-61f4-4554-b22b-d1ea723bbe00
                  job-name=prefect-k8s-example-8wn2j
Annotations:      <none>
Status:           Pending
IP:               10.1.0.9
IPs:
  IP:           10.1.0.9
Controlled By:  Job/prefect-k8s-example-8wn2j
Containers:
  prefect-job:
    Container ID:  
    Image:         prefect-docker-example:latest
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      python
      -m
      prefect.engine
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      PREFECT_API_URL:        <https://api.prefect.cloud/api/accounts/1ce80bf4-595f-42af-9feb-85431f7631a7/workspaces/2f7cd4bc-77a6-4b3b-8d68-dca0941eaae5>
      PREFECT_API_KEY:        @@@@@@@@@@@@
      PREFECT_LOGGING_LEVEL:  DEBUG
      PREFECT__FLOW_RUN_ID:   3e6cb334106a40de816ea42b40ba31b7
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2vxvl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-2vxvl:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type    Reason   Age                     From     Message
  ----    ------   ----                    ----     -------
  Normal  BackOff  3m57s (x1519 over 19h)  kubelet  Back-off pulling image "prefect-docker-example:latest"
c
Here’s your image: 
prefect-docker-example:latest
that’s not a valid image on dockerhub
if you aren’t doing anything specific, or customized, you should just be using the base prefect image
j
in my Dockerfile I have 
FROM prefecthq/prefect:2-python3.10
Are you saying this is not a base prefect image?
c
It is a valid image , but where is that image going ? Your k8s block is referencing an image that doesn’t exist
The default behavior without a registry is to pull from public docker hub
Your image from what I can tell doesn’t exist as a public image on docker hub otherwise I’d be able to search for it
If you built it , you need to tag and push it , then use that for your k8s block
Where did you push the image that you build from your Dockerfile?
j
ok so what I did I created a repo and pushed the image into dockerhub https://hub.docker.com/r/jafarsaedi/prefect-docker-example
should I make a change to image section in the k8s block? right now it says : 
Image
prefect-docker-example:latest
I guess question is now that dockerhub has my image, how do I reference the dockerhub image in my deployment
c
You would reference it with image=
jafarsaedi/prefect-docker-example
docker hub is the registry, 
jafarsaedi
would be your repository, 
prefect-docker-example
would be the tag
without 
:latest
tagged at the end, a pull will always reference :latest
j
worked like a charm! thanks a lot
✅ 1
🙌 2
66 Views
Open in Slack
PreviousNext
Powered by