https://prefect.io logo
Title
s

Sean Turner

10/04/2022, 7:50 PM
Is it possible to specify separate
KubernetesJob
infra blocks in a
Deployment
? e.g.
flow a
  task b (uses KubernetesJob with docker image foo)
  task c (uses KubernetesJob with docker image bar)
It kind of seems like I'm limited to to having one
KubernetesJob
per
Deployment
?
1
m

Matthias

10/04/2022, 7:51 PM
That one seems like a nice feature! But perhaps it could be (made) possible on a subflow level? This would be such an awesome feature btw. I am using Argo Workflows all the time and I like the flexibility of being able to use different containers at a task level. But this also forces you into making bigger (less granular) tasks to avoid the overhead of starting too many containers.
👍 1
s

Sean Turner

10/04/2022, 7:56 PM
I think that could work, no? Would need to have separate deployment lifecycles which does add some complexity however?
deployment a (uses KubernetesJob with docker image foo)
  flow a
    task a 
    flow b (calls task b)
  
deployment b (uses KubernetesJob with docker image bar)
  flow b
    task b
Would be nice if the
flow
or
task
decorators had arguments that overwrote the
infra block
specified in the
deployment
m

Matthias

10/04/2022, 8:00 PM
Infra is defined on a flow level (not on a task level). So I think it is possible to already do that today, but I guess you would need to create deployments for each subflow and create a main flow would then trigger these subflows (which was already possible in v1.x btw). But it would be nice if we could avoid creating separate deployments for each subflow…
s

Sean Turner

10/04/2022, 8:03 PM
Infra is defined on a flow level (not on a task level).
I think that is a bit of a misnomer with my diagram, will edit it to make it more clear.
a

Anna Geller

10/04/2022, 9:08 PM
What a timing! We were just discussing this. Not to promise anything too early but I believe we'll have better support for that in the future
🎉 3