Nathan
09/17/2023, 7:53 PMFileNotFoundError
every single time a flow is triggered (tracebacks in the comments). I know for a fact these files/directories exist because I shell into the pods and verify it. Fair to say it's driving me nuts.
Here's how I'm set up:
1. Official Prefect Helm chart >> https://github.com/PrefectHQ/prefect-helm/tree/main/charts/prefect-worker
a. Nothing modified except the values it asks for.
2. Official Docker image >> prefecthq/prefect:2.12.0-python3.11-kubernetes
3. I have a flows
directory with sub-modules for each flow
4. Alongside the flows
directory, I have two standalone python files that build Blocks using KubernetesJob()
and .save()
and deploy Flows using deployment.Build_from_flow()
.
a. In the Dockerfile, I just copy all of this into /opt/prefect
so resulting directory is /opt/prefect/flows/, blocks.py, deploy.py
5. I have a custom entrypoint.sh
script that logs into Prefect, runs the files to build blocks and deploy flows, and then starts the worker. It does not modify file structure at all and everything runs as expected, i.e. Blocks built, Flows deployed, and Worker started in k8s.
Any help or pointers as to what is going wrong would be greatly appreciated. I've been trying to debug this for a few days now.Deployment.build_from_flow()
func. I import all of my modules and then loop through them and deploy them this way:
deployed_flow = Deployment.build_from_flow(
flow=deployment,
name=deployment.NAME, schedule=deployment.SCHEDULE, infrastructure=job_infra, path=f"/opt/prefect/flows/{deployment}",
skip_upload=True,
apply=True,
work_pool_name="my-work-pool",
work_queue_name="my-work-queue",
)
This works as intended and shows as such in the Cloud UI.Eric Sales De Andrade
09/21/2023, 10:14 AM