Hello I m working on deploying a Prefect worker to a Kuberne Prefect Community #prefect-kubernetes

Hello! I'm working on deploying a Prefect worker ...

Nathan

09/17/2023, 7:53 PM

Hello! I'm working on deploying a Prefect worker to a Kubernetes cluster. However, I get

FileNotFoundError

every single time a flow is triggered (tracebacks in the comments). I know for a fact these files/directories exist because I shell into the pods and verify it. Fair to say it's driving me nuts. Here's how I'm set up: 1. Official Prefect Helm chart >> https://github.com/PrefectHQ/prefect-helm/tree/main/charts/prefect-worker a. Nothing modified except the values it asks for. 2. Official Docker image >>

prefecthq/prefect:2.12.0-python3.11-kubernetes

3. I have a

flows

directory with sub-modules for each flow 4. Alongside the

flows

directory, I have two standalone python files that build Blocks using

KubernetesJob()

and

.save()

and deploy Flows using

deployment.Build_from_flow()

. a. In the Dockerfile, I just copy all of this into

/opt/prefect

so resulting directory is

/opt/prefect/flows/, blocks.py, deploy.py

5. I have a custom

entrypoint.sh

script that logs into Prefect, runs the files to build blocks and deploy flows, and then starts the worker. It does not modify file structure at all and everything runs as expected, i.e. Blocks built, Flows deployed, and Worker started in k8s. Any help or pointers as to what is going wrong would be greatly appreciated. I've been trying to debug this for a few days now.

Nathan

09/17/2023, 7:54 PM

I've played a bit with file structure when debugging, but it always errors out to one of these two tbs:

Nathan

09/17/2023, 7:58 PM

This is my

Deployment.build_from_flow()

func. I import all of my modules and then loop through them and deploy them this way:

Copy code

deployed_flow = Deployment.build_from_flow(
flow=deployment,
name=deployment.NAME, schedule=deployment.SCHEDULE, infrastructure=job_infra, path=f"/opt/prefect/flows/{deployment}", 
skip_upload=True,
apply=True,
work_pool_name="my-work-pool",
work_queue_name="my-work-queue",
)

This works as intended and shows as such in the Cloud UI.

Eric Sales De Andrade

09/21/2023, 10:14 AM

Hi @Nathan, did you manage to figure this out? Having the same issue and can’t figure it out for the life of me

Olli Kavén

10/03/2023, 12:47 PM

@Eric Sales De Andrade or @Nathan did you figure this out? We run into the same issue

Eric Sales De Andrade

10/03/2023, 12:56 PM

Hey I ended up pulling the flows from Github instead using a Secret Block

Olli Kavén

10/03/2023, 1:24 PM

Solved this. In our test we were using

latest

tag in the images. That caused the images to be cached in kubernetes. We tagged the images with unique id and put that id in the

image

job_variables

. After that the correct image was pulled. We also changed the path from

/opt/prefect

/prefect_temp

but not sure if that had any impact.

Eric Sales De Andrade

10/03/2023, 2:43 PM

thanks that’s great. I had the worker image policy set on

AlwaysPull

but didn’t know the tag caused this issue too. Thanks for the recommendation thank you

👍 1

Nathan

10/26/2023, 8:36 PM

Hi fellas. Sorry for late response. The fix for me was not using a kubernetes infra block in the first place when I was using the Prefect's worker model. The infrastructure blocks are for the agent model, not the worker model

Open in Slack

Previous Next