Nathan
09/18/2023, 12:33 PMprefecthq/prefect:2.12.0-python3.11-kubernetes
image and a standalone Python script that loops through my flows and deploys them using Deployment.build_from_flow()
. I've tried multiple file setups and always get some variation of FileNotFound
error when a flow is triggered (the deployments themselves work and show in the Cloud UI as intended).
Feels like there is a simple solution and I've just missed it somewhere. If it was clear to me how Prefect looks for flows then I can just set it up that way, so any insight is appreciated.
More technical explanation of my setup can be found in #prefect-kubernetes >> https://prefect-community.slack.com/archives/C048SVCEFF0/p1694980409534619.Christopher Boyd
09/18/2023, 4:35 PM/opt/prefect
- you can use path and entrypoint to set location inside the container, or if you’re using workers
your entrypoint + set working directorymain.py:hello_world
, then the default location would be /opt/prefect/main.py
if you put it somehwere else, like /my/custom/path
, then you would set either your path or working directory to that prefect.deployment.steps.set_working_directory
and your entrypoint wouldn’t changeNathan
09/18/2023, 4:46 PM/opt/prefect/flows/hello_world/hello_world.py:hello_world_flow
, then the entrypoint would just be hello_world.py:hello_world_flow
and the path would be everything else up to that?Christopher Boyd
09/18/2023, 4:53 PMNathan
09/18/2023, 4:54 PMpath=/opt/prefect/flows/hello_world
and entrypoint=hello_world.py:hello_world_flow
. Everything stands up fine and shows in the UI but when the flow is triggered, I get the below error. WORKDIR = /opt/prefect
.Christopher Boyd
09/18/2023, 6:43 PMNathan
09/18/2023, 6:54 PMkubectl exec
into the pod and walk the filetree.Christopher Boyd
09/18/2023, 6:54 PMNathan
09/18/2023, 7:03 PM/opt/prefect/deploy.py
imports all of the submodules from flows
directory, loops through them, and uses Deployment.build_from_flow()
to deploy each of them.
Deployment:
deployed_flow = Deployment.build_from_flow(
flow=deployment, # hello_world_flow imported from flows/hello_world/hello_world.py
name=deployment.NAME, # "Hello, world!"
schedule=deployment.SCHEDULE, # CronSchedule(cron="0 9 * * *", timezone="UTC")
infrastructure=job_infra, # KubernetesJob.load("small-k8s-job")
path=deployment_path, # "/opt/prefect/flows/hello_world/"
entrypoint=entrypoint, # "hello_world.py:hello_world_flow"
skip_upload=True,
apply=True,
work_pool_name="my-work-pool",
work_queue_name="k8s",
)
Kubernetes block (which is loaded above) looks like this:
k8s_config = KubernetesJob(
namespace="my-namespace",
finished_job_ttl=60,
job_watch_timeout_seconds=None,
pod_watch_timeout_seconds=300,
image=PREFECT_IMAGE, # prefecthq/prefect:2.12.0-python3.11-kubernetes with our flows and requirements.txt added
image_pull_policy=KubernetesImagePullPolicy.IF_NOT_PRESENT,
customizations=job_infra[1], # K8s pod resource customizations
)
k8s_config.save(...)
Christopher Boyd
09/18/2023, 7:34 PMdeployment = Deployment.build_from_flow(
flow=flow,
name=deployment_name,
work_pool_name=f"{my work pool}",
work_queue_name="default",
infra_overrides=infra_overrides,
path="/my/custom/path",
entrypoint="flow.py:{flow_name}
)
Nathan
09/18/2023, 7:37 PMSMALL_K8S_INFRA = [
{
"op": "add",
"path": "/spec/template/spec/containers/0/resources",
"value": {
"limits": {"cpu": "500m", "memory": "512Mi", "ephemeral-storage": "5Gi"},
"requests": {"cpu": "100m", "memory": "128Mi", "ephemeral-storage": "1Gi"},
},
}
]
I could use your code and plug in the above for infra_overrides
?Christopher Boyd
09/18/2023, 7:45 PMinfra_overrides = {
"image": image,
"env": environment,
"cpu_limit": "1",
"memory_limit": "6Gi"
}
You’d then need to surface cpu_limit
and memory_limit
as top level items in the work pool. (See pictures)cpu_limit
as an overrideNathan
09/18/2023, 7:48 PMChristopher Boyd
09/18/2023, 7:50 PMNathan
09/18/2023, 7:52 PM