Blake Hamm
08/18/2022, 10:48 PMDeployment
class. Previously, I defined a KubernetesJob
infrastructure block in a different python script and used the .save()
method to instantiate a block in the UI. It looks like this was removed in 2.1... How can I do this now?
Moreover, I'm trying to understand what I pass in for the Deployment
class in the infrastructure
parameter. With the storage
parameter, I just load the storage
block that I previously defined. Can I do the same with the infrastructure
block? If so, how do I load an existing KubernetesJob
block? Or, is the new workflow to just input the KubernetesJob
dictionary directly in your Deployment
class? If that's the case, how do I tie this to a work queue?
In general, I'm a big fan of the KubernetesJob
block. idc whether it's an actual block or just a python dictionary I pass into the deployment. Either way, it's been extremely helpful to manage compute depending on the deployment. Would love to streamline it with CICD. I know that's in the works and I'm eager to see best practices with this and implement it.Anna Geller
08/19/2022, 12:09 AMKubernetesJob
block as you do with the storage block. it would be:
KubernetesJob.load("name")
I think for CI/CD, the CLI experience is much easier, but up to youIlya Galperin
08/19/2022, 12:40 AMKubernetesJob blocks
Anna or is the recommended method to use blocks for more advanced use cases? https://github.com/anna-geller/dataflow-ops/blob/main/blocks/kubernetes-job/infra_from_yaml_manifest.pyAnna Geller
08/19/2022, 1:51 AMk8s_job.save("yourname", overwrite=True)
Ilya Galperin
08/19/2022, 2:17 AMKubernetesJob
, saving it as a block to the local file system that is running the pipeline, then passing that block to the --infra-block
flag in the CLI when running deployment build
?Anna Geller
08/19/2022, 2:29 AM--override image=xxx
Blake Hamm
08/19/2022, 1:35 PM