https://prefect.io logo
t

Thomas Fredriksen

09/30/2022, 8:16 AM
Hi there, I was wondering if there are any arguments against running an Orion deployment in kubernetes with a single agent? If I only wish for the jobs to run as KubernetesJobs, it seems to me that having a single agent with an appropriate service account is the only thing necessary to have an effective deployment? Furthermore - Since we only need a single agent, could we simply have the agent run as a sidecar to the orion server? I guess we would have to disable pod autoscalers to avoid any issues that may arrise when the number of servers (and thus agents) decrease?
z

Zanie

09/30/2022, 3:36 PM
That’s a great deployment pattern!
Originally we used to run the agent/server in a single container, even. We split it out for flexibility. The only downside is that if the agent picks up work that runs locally like a
Process
it could cause strain on your server’s pod.
It’s safer to have them deployed separately, but you can definitely configure it to avoid that situation.
I’m not sure I follow the concerns re scaling / sidecars.
t

Thomas Fredriksen

09/30/2022, 4:08 PM
Cool, that’s nice to know :)
I guess it depends on how the HPA is configured, but what happens to the jobs an agent is running if the if the deployment is suddenly scaled down?
k

Kevin Wang

10/03/2022, 3:29 PM
Related to Thomas's question. Are the 'retries' on a Flow run sent back to the work queue, prior to the next retry? I'm seeing my flow run become stuck on 'Running'' (when my agent running the local process gets shut down). Then, it never gets tried by another agent. Is this expected.. if flow runs don't have a timeout, are they considered 'lost' in a queue, if an agent shuts down mid-run?
🙌 1
👀 1
6 Views