https://prefect.io logo
n

Nathan

09/17/2023, 7:53 PM
Hello! I'm working on deploying a Prefect worker to a Kubernetes cluster. However, I get
FileNotFoundError
every single time a flow is triggered (tracebacks in the comments). I know for a fact these files/directories exist because I shell into the pods and verify it. Fair to say it's driving me nuts. Here's how I'm set up: 1. Official Prefect Helm chart >> https://github.com/PrefectHQ/prefect-helm/tree/main/charts/prefect-worker a. Nothing modified except the values it asks for. 2. Official Docker image >>
prefecthq/prefect:2.12.0-python3.11-kubernetes
3. I have a
flows
directory with sub-modules for each flow 4. Alongside the
flows
directory, I have two standalone python files that build Blocks using
KubernetesJob()
and
.save()
and deploy Flows using
deployment.Build_from_flow()
. a. In the Dockerfile, I just copy all of this into
/opt/prefect
so resulting directory is
/opt/prefect/flows/, blocks.py, deploy.py
5. I have a custom
entrypoint.sh
script that logs into Prefect, runs the files to build blocks and deploy flows, and then starts the worker. It does not modify file structure at all and everything runs as expected, i.e. Blocks built, Flows deployed, and Worker started in k8s. Any help or pointers as to what is going wrong would be greatly appreciated. I've been trying to debug this for a few days now.
I've played a bit with file structure when debugging, but it always errors out to one of these two tbs:
This is my
Deployment.build_from_flow()
func. I import all of my modules and then loop through them and deploy them this way:
Copy code
deployed_flow = Deployment.build_from_flow(
flow=deployment,
name=deployment.NAME, schedule=deployment.SCHEDULE, infrastructure=job_infra, path=f"/opt/prefect/flows/{deployment}", 
skip_upload=True,
apply=True,
work_pool_name="my-work-pool",
work_queue_name="my-work-queue",
)
This works as intended and shows as such in the Cloud UI.
e

Eric Sales De Andrade

09/21/2023, 10:14 AM
Hi @Nathan, did you manage to figure this out? Having the same issue and can’t figure it out for the life of me