Is there a downside to running a worker to dispatch to a kubernetes work pool in the same pod as the prefect server? I'm running on a GKE autopilot cluster that requires significant mimimum resources that should support both processes.
n
Nate
02/19/2024, 10:06 PM
hi @Sean Davis - can you expand on this a bit?
> running a worker to dispatch to a kubernetes work pool
typically a worker runs in its own pod and picks up work from / gets a job infra template from a work pool
s
Sean Davis
02/20/2024, 12:17 AM
If I were on a non-autopilot cluster, I would definitely run the worker in its own pod. In the context of an autopilot cluster, I'm trying to avoid running two pods since the pod minimum requirements are significant for the workload (a few flows/day for the worker, 1 user for the server). This is purely about utilizing resources efficiently.
Sean Davis
02/20/2024, 12:19 AM
Basically, I'd just stack
prefect server start
and
prefect worker start --pool kube-pool
. I don't see much downside for my simple system since scaling won't be needed on either the server or the worker. Just thought I'd ask.
n
Nate
02/20/2024, 12:21 AM
gotcha!
> simple system since scaling won't be needed on either the server or the worker
for this context this seems reasonable to me 👍
but im not a GKE autopilot expert 🙂
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.