Charlie Nash
10/21/2025, 5:36 PMPID 34 stream output:
/tmp/prefect-fp3qhzpm.sh: line 1: kubectl: command not foundMarvin
10/21/2025, 5:36 PMMarvin
10/21/2025, 5:39 PMkubectl, wrote it into a temporary script at /tmp/prefect-*.sh, and the environment where it ran does not have kubectl installed (or it’s not on PATH). That’s why you see:
/tmp/prefect-fp3qhzpm.sh: line 1: kubectl: command not found
Why Prefect is creating /tmp/prefect-*.sh
- Prefect executes shell commands by writing them to a temp script and running it. This happens when you:
- Use a shell “operation”/task (from the Prefect shell integration)
- Use a deployment step that runs a shell script (e.g., in prefect.yaml build steps)
Common scenarios and fixes
- Process work pool (runs directly on a VM/laptop/server):
- Fix: Install kubectl on the machine running the worker and ensure it’s on PATH.
- Quick check: run which kubectl on that host.
- Docker work pool:
- Fix: Your flow runs inside a container; the image must include kubectl. Prefect’s base images don’t include it by default. Use a custom image or one that already has kubectl.
- Example Dockerfile snippet (Debian/Ubuntu base):
FROM prefecthq/prefect:3-latest
RUN apt-get update && apt-get install -y curl ca-certificates && \
curl -sSL -o /usr/local/bin/kubectl <https://dl.k8s.io/release/v1.30.0/bin/linux/amd64/kubectl> && \
chmod +x /usr/local/bin/kubectl
- Example for Alpine:
FROM prefecthq/prefect:3-latest
RUN apk add --no-cache kubectl
- Kubernetes work pool:
- Fix: Same idea—your pod’s image must include kubectl if your flow or shell step invokes it. Update the work pool’s job template/image accordingly.
- Deployment build steps (run on your local/dev machine during prefect deploy):
- If the error happens while building a deployment (before the flow runs), install kubectl on the machine where you run the deploy command, or remove that step.
PATH considerations
- If kubectl is installed but not on PATH, either:
- Use the full path to the binary in your command (e.g., /usr/local/bin/kubectl ...), or
- Add it to PATH in the environment where the command runs.
Helpful docs
- Shell integration: Prefect Shell integration
- Docker work pools: Docker work pool guide
- Kubernetes work pools: Kubernetes work pool guide
- Work pool/job template config: Manage work pools
- Customizing job variables (image, env): Customize job variables
To help pinpoint the fix, could you share:
- Which work pool type you’re using (Process, Docker, or Kubernetes)?
- Where the kubectl call is coming from (a shell task/operation in your flow vs. a deployment build step)?
- The image you’re using (if Docker/Kubernetes) or the host OS (if Process)?Charlie Nash
10/21/2025, 5:48 PMPID 34 stream output:
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:archtics-uc:default" cannot list resource "pods" in API group "" at the cluster scopeMarvin
10/21/2025, 5:48 PMMarvin
10/21/2025, 5:48 PMkubectl get pods with `-A`/all-namespaces or similar) tried to list Pods at the cluster scope, but the identity in use is the namespace’s default service account:
system:serviceaccount:archtics-uc:default
That service account does not have permission to list Pods cluster-wide, so the API returns Forbidden.
How to fix it
Pick the least-privileged option that matches what you actually need:
1) If you only need Pods in the archtics-uc namespace
- Change your command to target the namespace:
kubectl get pods -n archtics-uc
- Grant namespaced permissions to that service account:
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
namespace: archtics-uc
name: list-pods
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get","list","watch"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
name: list-pods-binding
namespace: archtics-uc
subjects:
- kind: ServiceAccount
name: default
namespace: archtics-uc
roleRef:
kind: Role
name: list-pods
apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
2) If you truly need cluster-wide access (e.g., -A)
- Grant cluster-wide RBAC to the service account:
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
name: list-pods-clusterwide
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get","list","watch"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRoleBinding
metadata:
name: list-pods-clusterwide-binding
subjects:
- kind: ServiceAccount
name: default
namespace: archtics-uc
roleRef:
kind: ClusterRole
name: list-pods-clusterwide
apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
Prefect-specific notes
- If this is running in a Prefect Kubernetes work pool, your flow runs inside a pod that uses a service account. If you didn’t set one explicitly, it’s usually the namespace’s default SA (as shown in your error). You can:
- Create a dedicated service account with the needed permissions (recommended), and
- Set serviceAccountName in your work pool’s base job template or job/pod spec so Prefect jobs use that SA.
- Alternatively, narrow the kubectl commands to your namespace and grant only namespaced permissions.
Helpful checks
- See what the SA can do:
kubectl auth can-i list pods --as=system:serviceaccount:archtics-uc:default -n archtics-uc
kubectl auth can-i list pods --as=system:serviceaccount:archtics-uc:default -A
References
- Prefect Kubernetes work pool guide: Kubernetes work pool guide
- Kubernetes RBAC overview: K8s RBAC docs
- Checking access: kubectl auth can-i
If you can share how you’re running the command (shell task in a flow vs. something in a deployment step) and which work pool you’re using, I can suggest the exact place to set serviceAccountName or adjust the command.Charlie Nash
10/21/2025, 5:55 PMPID 33 stream output:
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:archtics-uc:default" cannot list resource "pods" in API group "" in the namespace "archtics-uc"Marvin
10/21/2025, 5:55 PMMarvin
10/21/2025, 5:59 PMapiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
namespace: archtics-uc
name: list-pods
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get","list","watch"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
name: list-pods-binding
namespace: archtics-uc
subjects:
- kind: ServiceAccount
name: default
namespace: archtics-uc
roleRef:
kind: Role
name: list-pods
apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
B) Recommended: Create a dedicated SA and bind it
- Use a separate service account and bind only what you need:
apiVersion: v1
kind: ServiceAccount
metadata:
name: archtics-uc-prefect
namespace: archtics-uc
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
namespace: archtics-uc
name: list-pods
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get","list","watch"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
name: list-pods-binding
namespace: archtics-uc
subjects:
- kind: ServiceAccount
name: archtics-uc-prefect
namespace: archtics-uc
roleRef:
kind: Role
name: list-pods
apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
Prefect-specific step (Kubernetes work pool)
- Make sure your Prefect jobs use that service account. In a Kubernetes work pool, set serviceAccountName in the pod spec of the base job template (path: spec.template.spec.serviceAccountName in the Job manifest). Then update the work pool to use that template:
prefect work-pool update <your-pool-name> --base-job-template path/to/template.json
Docs: Kubernetes work pool guide, Manage work pools
Validate permissions
- Before re-running the flow, check access:
kubectl auth can-i list pods -n archtics-uc --as=system:serviceaccount:archtics-uc:default
kubectl auth can-i list pods -n archtics-uc --as=system:serviceaccount:archtics-uc:archtics-uc-prefect
References
- Kubernetes RBAC overview: K8s RBAC docs
- Checking API access: kubectl auth can-i
If you share your work pool name and whether you’d like to use a dedicated SA, I can sketch the exact base job template snippet to set serviceAccountName in your pool.