Hello I ve had good success deploying with prefect in subpro

Question

Hello I ve had good success deploying with prefect in subprocess docker locally and also with ECR Now when it comes to k8s docker desktop k8s is running green with k8s POD prefect k8s running but prefect says Pod never started I followed a post by < Jeff Hale> and this is my build `prefect deployment build flows main py ingest everflow ` ` n prefect k8s example q k8s ` ` sb s3 s3 everflow ef testing docker k8s image ` ` ib kubernetes job prefect k8s example ` ` apply` ```14 40 49 825 | INFO | prefect agent Submitting flow run 99f81791 abda 488f 802e d949535cb351 Users jafaramin 1 env lib python3 10 site packages prefect agent py 216 UserWarning Block document has schema checksum sha256 1a553852c1bcc9cd95645917bdbaaae9e28cbcc1270ff5b43b45bbd87564f5cc which does not match the schema checksum for class KubernetesJob This indicates the schema has changed and this block may not load infrastructure block = Block from block document infra document 14 40 50 538 | INFO | prefect agent Completed submission of flow run 99f81791 abda 488f 802e d949535cb351 14 40 50 546 | DEBUG | prefect infrastructure kubernetes job Job prefect k8s example mwmgm Starting watch for pod start 14 40 50 555 | INFO | prefect infrastructure kubernetes job Job prefect k8s example mwmgm Pod has status Pending 14 40 54 827 | DEBUG | prefect agent Checking for flow runs 14 40 59 925 | DEBUG | prefect agent Checking for flow runs 14 42 50 560 | ERROR | prefect infrastructure kubernetes job Job prefect k8s example mwmgm Pod never started ```

Jeff Hale · Answer

The exact same deployment except for changing the infrastructure block works okay with Docker locally but not K8s locally What if you use ` i` with extra pip packages environment variable to run K8s on the latest Docker image does that work The only other thing that stands out to me that might be related is that your storage block slug is s3 s3 everflow ef testing docker k8s image is that the slug that comes up if you do `prefect block ls`

Jafar A · Answer

< Jeff Hale> yes same image of packages local or pulled from ecr runs ok with docker block but not working with k8s block trying adding i and quite frankly this is my first k8s experience so not much previous knowledge and I m getting error asking to install other packages pandas ect `prefect deployment build flows main py ingest everflow n prefect k8s example q k8s sb s3 s3 everflow ef testing docker k8s image i kubernetes job override env EXTRA PIP PACKAGES=s3fs apply` yes `s3 s3 everflow` is the slug and added some subfolder for s3 and I think it s optional i wanted a copy of code being sent to s3 folder I tried removing sb from deployment and still got Pod never started error

Jeff Hale · Answer

Hmm If same thing worked with Docker it is probably an issue with your k8s running locally Do kubectl commands work

Jafar A · Answer

kubectl still works `kubectl get all A` `NAMESPACE NAME READY STATUS RESTARTS AGE` `default pod organic foxhound xsl22 2tc2s 0 1 Completed 0 15m` `default pod prefect k8s example 8wn2j rzj72 0 1 ImagePullBackOff 0 10m` `default pod prefect k8s example d5mhb mcpsx 0 1 ImagePullBackOff 0 107m` `default pod prefect k8s example ecr 9wlmj 7fd9z 0 1 ImagePullBackOff 0 2m34s` `default pod prefect k8s example mwmgm j5mtx 0 1 ImagePullBackOff 0 97m`

Jeff Hale · Answer

Are there any logs in the containers in Docker Desktop

Jafar A · Answer

docker logs containerid returns logs for successful docker deployment when I try it on k8s container return none

Jeff Hale · Answer

hmm < Christopher Boyd> is a resident K8s expert Any thoughts here

Christopher Boyd · Answer

What do the following say ```kubectl get events kubectl describe pod prefect k8s example 8wn2j rzj72 ```

Christopher Boyd · Answer

ImagePullBackOff says it can t pull the image describe events should say why

Christopher Boyd · Answer

what image are you using

Jafar A · Answer

< Christopher Boyd> you re right and here is the output however description is not clear ``` env MacBook Pro de orchestration jamin$ kubectl get events LAST SEEN TYPE REASON OBJECT MESSAGE 3m40s Normal BackOff pod prefect k8s example 8wn2j rzj72 Back off pulling image prefect docker example latest 5m54s Normal Pulling pod prefect k8s example d5mhb mcpsx Pulling image prefect docker example latest 61s Normal BackOff pod prefect k8s example d5mhb mcpsx Back off pulling image prefect docker example latest 51s Normal BackOff pod prefect k8s example ecr 9wlmj 7fd9z Back off pulling image 72s Normal BackOff pod prefect k8s example mwmgm j5mtx Back off pulling image prefect docker example latest env MacBook Pro de orchestration jamin$ env MacBook Pro de orchestration jamin$ kubectl describe pod prefect k8s example 8wn2j rzj72 Name prefect k8s example 8wn2j rzj72 Namespace default Priority 0 Service Account default Node docker desktop 192 168 65 4 Start Time Thu 02 Mar 2023 16 08 19 0500 Labels controller uid=28d05e99 61f4 4554 b22b d1ea723bbe00 job name=prefect k8s example 8wn2j Annotations lt none gt Status Pending IP 10 1 0 9 IPs IP 10 1 0 9 Controlled By Job prefect k8s example 8wn2j Containers prefect job Container ID Image prefect docker example latest Image ID Port lt none gt Host Port lt none gt Args python m prefect engine State Waiting Reason ImagePullBackOff Ready False Restart Count 0 Environment PREFECT API URL PREFECT API KEY PREFECT LOGGING LEVEL DEBUG PREFECT FLOW RUN ID 3e6cb334106a40de816ea42b40ba31b7 Mounts var run secrets kubernetes io serviceaccount from kube api access 2vxvl ro Conditions Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes kube api access 2vxvl Type Projected a volume that contains injected data from multiple sources TokenExpirationSeconds 3607 ConfigMapName kube root ca crt ConfigMapOptional lt nil gt DownwardAPI true QoS Class BestEffort Node Selectors lt none gt Tolerations op=Exists for 300s op=Exists for 300s Events Type Reason Age From Message Normal BackOff 3m57s x1519 over 19h kubelet Back off pulling image prefect docker example latest ```

Christopher Boyd · Answer

Here s your image `prefect docker example latest`

Christopher Boyd · Answer

that s not a valid image on dockerhub

Christopher Boyd · Answer

if you aren t doing anything specific or customized you should just be using the base prefect image

Christopher Boyd · Answer

Jafar A · Answer

in my Dockerfile I have `FROM prefecthq prefect 2 python3 10` Are you saying this is not a base prefect image

Christopher Boyd · Answer

It is a valid image but where is that image going Your k8s block is referencing an image that doesn t exist

Christopher Boyd · Answer

The default behavior without a registry is to pull from public docker hub

Christopher Boyd · Answer

Your image from what I can tell doesn t exist as a public image on docker hub otherwise I d be able to search for it

Christopher Boyd · Answer

If you built it you need to tag and push it then use that for your k8s block

Christopher Boyd · Answer

Where did you push the image that you build from your Dockerfile

Jafar A · Answer

ok so what I did I created a repo and pushed the image into dockerhub <https hub docker com r jafarsaedi prefect docker example>

Jafar A · Answer

should I make a change to image section in the k8s block right now it says `Image` `prefect docker example latest` I guess question is now that dockerhub has my image how do I reference the dockerhub image in my deployment

Christopher Boyd · Answer

You would reference it with image=`jafarsaedi prefect docker example`

Christopher Boyd · Answer

docker hub is the registry `jafarsaedi` would be your repository `prefect docker example` would be the tag

Christopher Boyd · Answer

without ` latest` tagged at the end a pull will always reference latest

Jafar A · Answer

worked like a charm thanks a lot