<@ULVA73B9P> I'm running Prefect 3.4.4 OSS on Kube...
# ask-marvin
k
@Marvin I'm running Prefect 3.4.4 OSS on Kubernetes. When exactly do I need to use prefect-kubernetes package? What is the use case?
m
thought for 193.4 seconds
Great question. Short answer: you only need the prefect-kubernetes package when you want Prefect to create and manage Kubernetes Jobs for your flow runs or when you want to use Prefect’s Kubernetes helpers inside your flows. When you need prefect-kubernetes - You want each flow run to execute as its own Kubernetes Job/Pod with isolation and K8s-native scaling. - This uses the Kubernetes worker type and a Kubernetes work pool. - You want to customize job templates (image, namespace, resources, env, serviceAccount, volumes/mounts, tolerations, etc.) via a work pool’s base job template. - You want to use Prefect-provided tasks/blocks to interact with Kubernetes from flows (e.g., create/read jobs, pods, deployments, services; use a Kubernetes credentials/cluster config block). When you do NOT need it - You’re just running Prefect Server (OSS) on Kubernetes. - You run a Process worker inside a Kubernetes pod and are okay with flow runs being subprocesses of that single pod (no per-run K8s Jobs). - You use a different worker type (e.g., Docker, ECS, etc.) for execution. Typical setup with Kubernetes worker 1) Install the integration (on the machine/container that will run the worker):
Copy code
pip install -U prefect-kubernetes
2) Create a Kubernetes work pool and start a worker:
Copy code
prefect work-pool create "my-k8s-pool" --type kubernetes
prefect worker start --pool "my-k8s-pool"
3) Deploy your flow to that pool (Prefect 3.x):
Copy code
from prefect import flow

@flow
def my_flow():
    ...

my_flow.from_source("<https://github.com/org/repo>", entrypoint="path/to/flow.py:my_flow").deploy(
    name="k8s-deployment",
    work_pool_name="my-k8s-pool",
)
What the package provides - Kubernetes worker and job config: run each flow run as a Kubernetes Job with a customizable base job template. - Blocks/credentials: e.g.,
KubernetesClusterConfig
,
KubernetesCredentials
to configure cluster access. - Convenience tasks/flows for K8s resources: create/read/update Jobs, Pods, Deployments, Services, and Custom Objects from your Prefect flows. Docs and references - Prefect + Kubernetes integration: prefect-kubernetes docs - How to run on Kubernetes: Kubernetes worker/how-to - Workers concept: Workers overview - Helm chart (optional for deploying workers/server on K8s): prefect-helm If you can share how you want your flow runs to execute (per-run K8s Jobs vs. subprocess inside a single pod), I can recommend the exact setup and template options.
k
@Marvin Still don't get it. I run everything on kubernetes: Prefect Server and Prefect Worker. I can see that the worker successfully creates kubernetes jobs and pods for every flow run. What is the use case for prefect-kubernetes package?
m
thought for 45.1 seconds
Totally fair — if your worker is already creating Jobs/Pods, then you’re already using what prefect-kubernetes provides. What prefect-kubernetes is for - It’s the plugin that adds the Kubernetes worker type to Prefect. That worker is what turns each flow run into a Kubernetes Job/Pod. - It also bundles K8s-specific utilities you can call from flows (blocks for credentials/cluster config and tasks for Pods/Jobs/Deployments/Services/CRDs). Where you need it installed - Worker side: Required. The environment running
prefect worker start
for a Kubernetes work pool must have
prefect-kubernetes
(and the Kubernetes Python client). Otherwise it can’t submit Jobs to your cluster. - Flow image (the container that runs your code): Not required unless your flow code imports/uses
prefect_kubernetes.*
helpers. If your flow doesn’t call those, it only needs
prefect
(plus your app deps). When you wouldn’t need it - If you run a Process worker inside a pod (no per-run K8s Jobs), you don’t need prefect-kubernetes. You’re just running subprocesses inside that single pod. - Hosting Prefect Server on Kubernetes does not require it. How to tell in your setup - If you can create a work pool with
--type kubernetes
and see a worker submit Kubernetes Jobs for each flow run, then the worker environment already has
prefect-kubernetes
installed (either baked into the image or installed by your Helm/chart customization). Related links - Integration overview: prefect-kubernetes docs - How to run on Kubernetes: Kubernetes worker/how-to - Workers concept: Workers overview If helpful, share how your worker is deployed (image/Helm values), and I can point out exactly where prefect-kubernetes is being pulled in and whether you need it in your flow image too.
k
@Marvin My worker is deployed using the official Prefect Helm charts. That's what I have in my values file:
Copy code
worker:
  image:
    prefectTag: 3.4.4-python3.11-kubernetes
m
thought for 35.9 seconds
That tag tells the story. The Helm chart is running the “Kubernetes worker” image variant, which bundles the prefect-kubernetes integration: - Your image:
3.4.4-python3.11-kubernetes
= Prefect 3.4.4 + Python 3.11 + prefect-kubernetes installed. - That package is what gives the worker the ability to submit each flow run as a Kubernetes Job/Pod (the Kubernetes worker type), watch the Job/Pod, stream logs, and apply the work pool’s base job template. So in your setup, prefect-kubernetes is already being used by the worker. Without it, the worker could not be
--type kubernetes
and would not be able to create Jobs; you’d be limited to a Process worker (subprocess runs inside one pod). Where you need prefect-kubernetes vs. not: - Worker container: Required for Kubernetes workers (your tag already includes it). - Prefect Server: Not needed. - Flow run image: Only needed if your flow code imports/uses
prefect_kubernetes
helpers (e.g., create/read Pods/Jobs/Deployments from the flow or use the Kubernetes credentials/cluster config blocks). If you don’t import those, your flow image just needs Prefect + your app deps; the worker still creates the Job. If you ever change the worker image to a non-kubernetes variant or a custom image, ensure it has
prefect-kubernetes
(or
pip install prefect-kubernetes
) or the worker won’t be able to submit K8s Jobs. Helpful refs: - Integration overview: prefect-kubernetes docs - How to run on Kubernetes (work pools, base job template, etc.): Kubernetes worker/how-to - Helm chart: prefect-helm If you want, share your flow image details. I can confirm whether you need to add
prefect-kubernetes
there based on whether you’re calling any
prefect_kubernetes.*
utilities from your flows.