https://prefect.io logo
n

Nathan

09/18/2023, 12:33 PM
Hello -- could someone shed some light on how Prefect expects flows to be found in a Docker container? I'm using the
prefecthq/prefect:2.12.0-python3.11-kubernetes
image and a standalone Python script that loops through my flows and deploys them using
Deployment.build_from_flow()
. I've tried multiple file setups and always get some variation of
FileNotFound
error when a flow is triggered (the deployments themselves work and show in the Cloud UI as intended). Feels like there is a simple solution and I've just missed it somewhere. If it was clear to me how Prefect looks for flows then I can just set it up that way, so any insight is appreciated. More technical explanation of my setup can be found in #prefect-kubernetes >> https://prefect-community.slack.com/archives/C048SVCEFF0/p1694980409534619.
c

Christopher Boyd

09/18/2023, 4:35 PM
The default working directory is
/opt/prefect
- you can use path and entrypoint to set location inside the container, or if you’re using
workers
your entrypoint + set working directory
if you put your entrypoint as
main.py:hello_world
, then the default location would be
/opt/prefect/main.py
if you put it somehwere else, like
/my/custom/path
, then you would set either your path or working directory to that
prefect.deployment.steps.set_working_directory
and your entrypoint wouldn’t change
n

Nathan

09/18/2023, 4:46 PM
Thanks for the response, Christopher. So in my situation, if I have flow at
/opt/prefect/flows/hello_world/hello_world.py:hello_world_flow
, then the entrypoint would just be
hello_world.py:hello_world_flow
and the path would be everything else up to that?
c

Christopher Boyd

09/18/2023, 4:53 PM
That’s right
n

Nathan

09/18/2023, 4:54 PM
That's simple enough! Thank you
Still didn't work. This is with
path=/opt/prefect/flows/hello_world
and
entrypoint=hello_world.py:hello_world_flow
. Everything stands up fine and shows in the UI but when the flow is triggered, I get the below error.
WORKDIR = /opt/prefect
.
c

Christopher Boyd

09/18/2023, 6:43 PM
Does that exist in the image
Is /opt/prefect/flows/hello_world a path inside the image ?
n

Nathan

09/18/2023, 6:54 PM
Yes it is. Here's when I
kubectl exec
into the pod and walk the filetree.
c

Christopher Boyd

09/18/2023, 6:54 PM
what does your deploy code look like?
n

Nathan

09/18/2023, 7:03 PM
/opt/prefect/deploy.py
imports all of the submodules from
flows
directory, loops through them, and uses
Deployment.build_from_flow()
to deploy each of them. Deployment:
Copy code
deployed_flow = Deployment.build_from_flow(
                flow=deployment,  # hello_world_flow imported from flows/hello_world/hello_world.py
                name=deployment.NAME,  # "Hello, world!"
                schedule=deployment.SCHEDULE,  # CronSchedule(cron="0 9 * * *", timezone="UTC")
                infrastructure=job_infra,  # KubernetesJob.load("small-k8s-job")
                path=deployment_path,  # "/opt/prefect/flows/hello_world/"
                entrypoint=entrypoint,  # "hello_world.py:hello_world_flow"
                skip_upload=True,
                apply=True,
                work_pool_name="my-work-pool",
                work_queue_name="k8s",
            )
Kubernetes block (which is loaded above) looks like this:
Copy code
k8s_config = KubernetesJob(
        namespace="my-namespace",
        finished_job_ttl=60,
        job_watch_timeout_seconds=None,
        pod_watch_timeout_seconds=300,
        image=PREFECT_IMAGE,  # prefecthq/prefect:2.12.0-python3.11-kubernetes with our flows and requirements.txt added
        image_pull_policy=KubernetesImagePullPolicy.IF_NOT_PRESENT,
        customizations=job_infra[1],  # K8s pod resource customizations
    )

k8s_config.save(...)
c

Christopher Boyd

09/18/2023, 7:34 PM
are you using prefect workers then? Kubernetes blocks for infrastructure would be for agents
if you’re using workers, then you don’t need a kubernetes block for your configuration, it would be configured as part of the work pool
I just have something like:
Copy code
deployment = Deployment.build_from_flow(
    flow=flow,
    name=deployment_name,
    work_pool_name=f"{my work pool}",
    work_queue_name="default",
    infra_overrides=infra_overrides,
    path="/my/custom/path",
    entrypoint="flow.py:{flow_name}
)
n

Nathan

09/18/2023, 7:37 PM
Ah. I didn't know that. That seems like the likely issue then.
This is the k8s customizations I'm using:
Copy code
SMALL_K8S_INFRA = [
    {
        "op": "add",
        "path": "/spec/template/spec/containers/0/resources",
        "value": {
            "limits": {"cpu": "500m", "memory": "512Mi", "ephemeral-storage": "5Gi"},
            "requests": {"cpu": "100m", "memory": "128Mi", "ephemeral-storage": "1Gi"},
        },
    }
]
I could use your code and plug in the above for
infra_overrides
?
Btw, I'm new to Prefect so I appreciate your time helping with this.
c

Christopher Boyd

09/18/2023, 7:45 PM
so infra_overrides in this way, becomes mapped to top level elements of the work-pool. so my infra_overrides block looks like:
Copy code
infra_overrides = {
    "image": image,
    "env": environment,
    "cpu_limit": "1",
    "memory_limit": "6Gi"
}
You’d then need to surface
cpu_limit
and
memory_limit
as top level items in the work pool. (See pictures)
basically it’s setting up the config on the work pool directly, then you can override it per flow / deployment. So here I’m surfacing a “cpu_limit” variable, which allows me to pass
cpu_limit
as an override
if you didn’t want tod o it like that, you could pass it through as the “resources” patch like you have I think
n

Nathan

09/18/2023, 7:48 PM
Okay, great. That is all really helpful. I appreciate it!
c

Christopher Boyd

09/18/2023, 7:50 PM
the short is - workers move all job configuration directly to the work pool
so you can set up a basic / default work pool, expose variables you want to override per flow, then you can pass those through with infra_overrides
and then you don’t have to pass / attach a k8s block anymore at all
n

Nathan

09/18/2023, 7:52 PM
Got it. Thanks, Christopher.
🙌 1
Just made these changes and everything worked. I owe you one!