https://prefect.io logo
j

Joël Luijmes

11/23/2020, 1:01 PM
If I use a Kubernetes agent to run my flows, what would be the recommended way to execute non-python code? Spawn off new Kubernetes jobs in the cluster, or are there other ideas on this?
s

Scott Moreland

11/23/2020, 1:11 PM
Are you aware of the prefect shell command task? I also found it easy to write my own "command runner" task using python subprocess. Happy to share that here if useful.
j

Joël Luijmes

11/23/2020, 1:37 PM
Okay, but how does that work with the kubernetes agent? Then all the tooling should be present in the agent right? Or can we base the agent of a different container?
s

Scott Moreland

11/23/2020, 1:41 PM
Not familiar with Kubernetes, but if you use Kubernetes to start your prefect flow, you could have your prefect flow kickoff an arbitrary shell process. Your Kubernetes agent would only control how the prefect flow runs and prefect would manage the individual shell task within the flow. I'm perhaps missing some details about Kubernetes agents that are important to answering your question so please defer to anyone with more Kubernetes / Prefect experience.
j

Joël Luijmes

11/23/2020, 2:44 PM
Ah yea, but everything runs in containers in kubernetes. So spawning a shell, means it runs in the agent container. Hence my question. But thanks for thinking with me 🙂
s

Scott Moreland

11/23/2020, 2:46 PM
I see now, does Kubernetes have an API you call to spawn a process that gets distributed via the Kubernetes cluster?
j

Joël Luijmes

11/23/2020, 4:23 PM
Everything works through an API yes, but you usually interact with a cli tool called kubectl (or with a library, such as prefect). What you would do is create a job, which will execute on one of the available nodes.
s

Scott Moreland

11/23/2020, 4:29 PM
Thanks for the explanation. I'm evidently not going to be much help here. I'll be looking at setting up Prefect on Kubernetes in the near future, and I can let you know if I figure anything out that directly addresses your question.
k

Kyle Moon-Wright

11/23/2020, 4:52 PM
Hey @Joël Luijmes, I think it may depend on the type of work you’re doing - you can certainly execute with a ShellTask, however a Docker sidecar is always a good option.
j

Joël Luijmes

11/24/2020, 10:10 AM
Hmm seems as an interesting approach, but I think I find the off-spawning of another kubernetes job more clean
Thanks 🙂