https://prefect.io logo
Title
b

Blake Hamm

09/01/2022, 11:23 PM
Prefect 2.3 is exciting for sure! The speed of development is wild and I'm trying to keep up! I'm wondering if
KubernetesJob
blocks can interact with (the new)
DockerContainer
block. Is there a way to deploy with
--ib
as a
DockerContainer
as well as use the
KubernetesJob
? Or is there any roadmap to connect the two together (like how the
DockerContainer
block can access a
DockerRegistry
block? I imagine this isn't feasible, but generally, I really like the ability to pass in a manifest file to a
KubernetesJob
block. Specifically, I'm using EKS on AWS Fargate and really like the ability to define the resources. On the other hand the new
DockerContainer
block seems really handy to manage environments for specific flows. Right now I have one image on ECR with all the dependencies and it's much heavier than it needs to be. From a CI/CD perspective it would be great to have an action creating
DockerContainer
blocks based on the individual flows and another action creating
KubernetesJob
blocks based on different manifest files. In an ideal world, I would love a way to deploy using a
DockerContainer
block as the image inside the
KubernetesJob
block. A current (hacky) solution could be to loop through the flow-level docker files, register them to ECR based on their flow name and create all the necessary
KubernetesJob
blocks for each individual flow. I could use some kind of "resource" tag to pick the necessary manifest file. This would create a distinct
KubernetesJob
block for every flow even though I might only be using 3 distinct manifest files (just lot's of unique containers for each flow). This would also require an --sb (unlike the new standalone
DockerContainer
block.
1
m

Matt Conger

09/02/2022, 12:02 AM
Hey Blake! I think you may have already checked out this doc, but I think it would be a good reference. https://docs.prefect.io/api-ref/prefect/blocks/kubernetes/?h=from_file#prefect.blocks.kubernetes.KubernetesClusterConfig I think that solution would be a good method to explore and I can pass your ideas along to the team.
b

Blake Hamm

09/02/2022, 12:05 AM
Got it; yeah, I was just digging into it a bit and I felt like I was off track (maybe I wasn't). So the
KubernetesClusterConfig
is a good place to define the manifest files? And then I could essentially just use the
KubernetesJob
block and pull the
KubernetesClusterConfig
based on the resources I need?
👍 1
m

Matt Conger

09/02/2022, 9:14 PM
You can check out this video as well! That walks through setting up a Kubernetes Deployment

https://www.youtube.com/watch?v=WG9qphVJ028&ab_channel=Prefect