Hey all - we’re executing Prefect in k8s. We’re trying to figure out if we have anything to benefit from adding Workers to our setup. Our motivation is that we’d like to have work coming in as a webhook trigger, but its too frequent for us to want the agent to create a K8s Job for each one, so we thought it’d be nice if there was a worker that could receive this incoming traffic and handle it at its own leisure. Do K8s workers execute flows as distinct k8s jobs that are NOT the worker itself --- or do they execute it as processes internally?

they will create jobs yes - you can have an agent in k8s that executes flows as a process rather than a job

can’t we do the same for a worker? it sounds a bit counterintuitive to me - i’d expect the agent to launch jobs (distinct from itself) , and the worker to poll for work that it does itself (as an internal process) am i missing something? you’re saying it’s the other way around? workers do no work [themselves], and agents do (or can) ?

I still use an agent but from the docs the k8s worker creates jobs still https://prefecthq.github.io/prefect-kubernetes/worker/

so what do you use an agent for?

i’d expect the agent to launch jobs (distinct from itself) ,

and the worker to poll for work that it does itself (as an internal process)

these statements are true in some cases 🙂 agents do launch flow runs (jobs) , but agents are not strongly typed, so they look at the infra attached to a scheduled deployment's flow run and use that to decide how it should be submitted. if its a deployment with process type infra, it will run right there in the agent process. if its docker / k8s, it will try to submit the flow run as a container to a docker / k8s environment that it has access to. the worker does effectively the same thing, except there's no ambiguity about how the worker will submit the flow run, bc X type workers can only submit flow runs from X type work pools (e.g. X = Process, K8s, ECS etc) - workers also have extra features like understanding

pull

steps and can generally be thought of as next generation agents (that serve the same larger purpose as agents - i.e. submitting work to an execution environment)

it makes more sense to have (a) dedicated worker(s) that just runs each incoming flow as a process…

i agree with your intuition, you could easily have a process type worker than just picks up and runs the flow run itself when its triggered, as opposed to a k8s type worker that would submit that new work as a new job / pod