Blake Hamm
09/01/2022, 11:23 PMKubernetesJob
blocks can interact with (the new) DockerContainer
block. Is there a way to deploy with --ib
as a DockerContainer
as well as use the KubernetesJob
? Or is there any roadmap to connect the two together (like how the DockerContainer
block can access a DockerRegistry
block?
I imagine this isn't feasible, but generally, I really like the ability to pass in a manifest file to a KubernetesJob
block. Specifically, I'm using EKS on AWS Fargate and really like the ability to define the resources. On the other hand the new DockerContainer
block seems really handy to manage environments for specific flows. Right now I have one image on ECR with all the dependencies and it's much heavier than it needs to be.
From a CI/CD perspective it would be great to have an action creating DockerContainer
blocks based on the individual flows and another action creating KubernetesJob
blocks based on different manifest files. In an ideal world, I would love a way to deploy using a DockerContainer
block as the image inside the KubernetesJob
block.
A current (hacky) solution could be to loop through the flow-level docker files, register them to ECR based on their flow name and create all the necessary KubernetesJob
blocks for each individual flow. I could use some kind of "resource" tag to pick the necessary manifest file. This would create a distinct KubernetesJob
block for every flow even though I might only be using 3 distinct manifest files (just lot's of unique containers for each flow). This would also require an --sb (unlike the new standalone DockerContainer
block.Matt Conger
09/02/2022, 12:02 AMBlake Hamm
09/02/2022, 12:05 AMKubernetesClusterConfig
is a good place to define the manifest files? And then I could essentially just use the KubernetesJob
block and pull the KubernetesClusterConfig
based on the resources I need?