https://prefect.io logo
a

Alexandre Guitton

02/22/2023, 3:25 PM
Hello 👋 Quick question, if I want to run my flow on Kubernetes, I need to set up a
KubernetesJob block
and use that in my deployment. Now, let's say I have three different flows, which require different python dependencies. Does that mean I need to create my three Docker images, then create three blocks, one for each, and finally associate the right block with the right flow in my deployment?
d

Deceivious

02/22/2023, 3:55 PM
You can also use
infra_overrides
when creating a deployment where u can override the default docker image with a new manifest. That way a single kubernetes cluster [or prefect block ] can pull off different docker images.
z

Zanie

02/22/2023, 4:08 PM
You can also use
EXTRA_PIP_PACKAGES
with our official image to install things at runtime (although prebuilt images are generally more efficient)
1
✍️ 1
a

Alexandre Guitton

02/22/2023, 4:11 PM
If I understand correctly, my configuration will be as follows: • A
KubernetesJob
block giving the default configuration to trigger the k8s job. • An
S3 block
(to store my flow code) • An agent deployed somewhere. My workflow will be the following: • Implement my flow • Build and push the docker image containing the python dependencies. • Create a deployment for my flow by overwriting the infrastructure to give the correct docker image, service account, namespace etc. for the job + overwriting the s3 path for my flow code. • Apply my deployment And then when my deployment runs: • It enters the queue • the agent gets my flow, uses the KubernetesJob + overwitte to deploy a job to my Kubernetes cluster. • The job will create a Pod that will retrieve the code from S3 storage (with the S3 block + overwrite) and run the stream! Am I right?
2 Views