Alexandre Guitton
02/22/2023, 3:25 PMKubernetesJob block
and use that in my deployment.
Now, let's say I have three different flows, which require different python dependencies. Does that mean I need to create my three Docker images, then create three blocks, one for each, and finally associate the right block with the right flow in my deployment?Deceivious
02/22/2023, 3:55 PMinfra_overrides
when creating a deployment where u can override the default docker image with a new manifest. That way a single kubernetes cluster [or prefect block ] can pull off different docker images.Zanie
EXTRA_PIP_PACKAGES
with our official image to install things at runtime (although prebuilt images are generally more efficient)Alexandre Guitton
02/22/2023, 4:11 PMKubernetesJob
block giving the default configuration to trigger the k8s job.
• An S3 block
(to store my flow code)
• An agent deployed somewhere.
My workflow will be the following:
• Implement my flow
• Build and push the docker image containing the python dependencies.
• Create a deployment for my flow by overwriting the infrastructure to give the correct docker image, service account, namespace etc. for the job + overwriting the s3 path for my flow code.
• Apply my deployment
And then when my deployment runs:
• It enters the queue
• the agent gets my flow, uses the KubernetesJob + overwitte to deploy a job to my Kubernetes cluster.
• The job will create a Pod that will retrieve the code from S3 storage (with the S3 block + overwrite) and run the stream!
Am I right?