Hello -- could someone shed some light on how Prefect expects flows to be found in a Docker container? I'm using the

prefecthq/prefect:2.12.0-python3.11-kubernetes

image and a standalone Python script that loops through my flows and deploys them using

Deployment.build_from_flow()

. I've tried multiple file setups and always get some variation of

FileNotFound

error when a flow is triggered (the deployments themselves work and show in the Cloud UI as intended). Feels like there is a simple solution and I've just missed it somewhere. If it was clear to me how Prefect looks for flows then I can just set it up that way, so any insight is appreciated. More technical explanation of my setup can be found in #prefect-kubernetes >> https://prefect-community.slack.com/archives/C048SVCEFF0/p1694980409534619.

The default working directory is

/opt/prefect

- you can use path and entrypoint to set location inside the container, or if you’re using

workers

your entrypoint + set working directory

Thanks for the response, Christopher. So in my situation, if I have flow at

/opt/prefect/flows/hello_world/hello_world.py:hello_world_flow

, then the entrypoint would just be

hello_world.py:hello_world_flow

and the path would be everything else up to that?

That’s right

That's simple enough! Thank you

Does that exist in the image

Yes it is. Here's when I

kubectl exec

into the pod and walk the filetree.

what does your deploy code look like?

/opt/prefect/deploy.py

imports all of the submodules from

flows

directory, loops through them, and uses

Deployment.build_from_flow()

to deploy each of them. Deployment:

deployed_flow = Deployment.build_from_flow(
                flow=deployment,  # hello_world_flow imported from flows/hello_world/hello_world.py
                name=deployment.NAME,  # "Hello, world!"
                schedule=deployment.SCHEDULE,  # CronSchedule(cron="0 9 * * *", timezone="UTC")
                infrastructure=job_infra,  # KubernetesJob.load("small-k8s-job")
                path=deployment_path,  # "/opt/prefect/flows/hello_world/"
                entrypoint=entrypoint,  # "hello_world.py:hello_world_flow"
                skip_upload=True,
                apply=True,
                work_pool_name="my-work-pool",
                work_queue_name="k8s",
            )

Kubernetes block (which is loaded above) looks like this:

k8s_config = KubernetesJob(
        namespace="my-namespace",
        finished_job_ttl=60,
        job_watch_timeout_seconds=None,
        pod_watch_timeout_seconds=300,
        image=PREFECT_IMAGE,  # prefecthq/prefect:2.12.0-python3.11-kubernetes with our flows and requirements.txt added
        image_pull_policy=KubernetesImagePullPolicy.IF_NOT_PRESENT,
        customizations=job_infra[1],  # K8s pod resource customizations
    )

k8s_config.save(...)

are you using prefect workers then? Kubernetes blocks for infrastructure would be for agents

Ah. I didn't know that. That seems like the likely issue then.

so infra_overrides in this way, becomes mapped to top level elements of the work-pool. so my infra_overrides block looks like:

infra_overrides = {
    "image": image,
    "env": environment,
    "cpu_limit": "1",
    "memory_limit": "6Gi"
}

You’d then need to surface

cpu_limit

and

memory_limit

as top level items in the work pool. (See pictures)

Okay, great. That is all really helpful. I appreciate it!

the short is - workers move all job configuration directly to the work pool

Got it. Thanks, Christopher.