<@ULVA73B9P> Can you explain the following log err...
# ask-marvin
c
@Marvin Can you explain the following log error I received?
Copy code
PID 34 stream output:
/tmp/prefect-fp3qhzpm.sh: line 1: kubectl: command not found
m
thought for 187.1 seconds
Short version: Prefect tried to run a shell command that included
kubectl
, wrote it into a temporary script at
/tmp/prefect-*.sh
, and the environment where it ran does not have
kubectl
installed (or it’s not on
PATH
). That’s why you see:
/tmp/prefect-fp3qhzpm.sh: line 1: kubectl: command not found
Why Prefect is creating /tmp/prefect-*.sh - Prefect executes shell commands by writing them to a temp script and running it. This happens when you: - Use a shell “operation”/task (from the Prefect shell integration) - Use a deployment step that runs a shell script (e.g., in
prefect.yaml
build steps) Common scenarios and fixes - Process work pool (runs directly on a VM/laptop/server): - Fix: Install
kubectl
on the machine running the worker and ensure it’s on
PATH
. - Quick check: run
which kubectl
on that host. - Docker work pool: - Fix: Your flow runs inside a container; the image must include
kubectl
. Prefect’s base images don’t include it by default. Use a custom image or one that already has
kubectl
. - Example Dockerfile snippet (Debian/Ubuntu base):
Copy code
FROM prefecthq/prefect:3-latest
    RUN apt-get update && apt-get install -y curl ca-certificates && \
        curl -sSL -o /usr/local/bin/kubectl <https://dl.k8s.io/release/v1.30.0/bin/linux/amd64/kubectl> && \
        chmod +x /usr/local/bin/kubectl
- Example for Alpine:
Copy code
FROM prefecthq/prefect:3-latest
    RUN apk add --no-cache kubectl
- Kubernetes work pool: - Fix: Same idea—your pod’s image must include
kubectl
if your flow or shell step invokes it. Update the work pool’s job template/image accordingly. - Deployment build steps (run on your local/dev machine during
prefect deploy
): - If the error happens while building a deployment (before the flow runs), install
kubectl
on the machine where you run the deploy command, or remove that step. PATH considerations - If
kubectl
is installed but not on
PATH
, either: - Use the full path to the binary in your command (e.g.,
/usr/local/bin/kubectl ...
), or - Add it to
PATH
in the environment where the command runs. Helpful docs - Shell integration: Prefect Shell integration - Docker work pools: Docker work pool guide - Kubernetes work pools: Kubernetes work pool guide - Work pool/job template config: Manage work pools - Customizing job variables (image, env): Customize job variables To help pinpoint the fix, could you share: - Which work pool type you’re using (Process, Docker, or Kubernetes)? - Where the
kubectl
call is coming from (a shell task/operation in your flow vs. a deployment build step)? - The image you’re using (if Docker/Kubernetes) or the host OS (if Process)?
c
@Marvin Can you explain this error and how to fix it?
Copy code
PID 34 stream output:
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:archtics-uc:default" cannot list resource "pods" in API group "" at the cluster scope
m
thought for 29.3 seconds
This is a Kubernetes RBAC error. The command you ran (likely
kubectl get pods
with `-A`/all-namespaces or similar) tried to list Pods at the cluster scope, but the identity in use is the namespace’s default service account:
system:serviceaccount:archtics-uc:default
That service account does not have permission to list Pods cluster-wide, so the API returns Forbidden. How to fix it Pick the least-privileged option that matches what you actually need: 1) If you only need Pods in the archtics-uc namespace - Change your command to target the namespace:
Copy code
kubectl get pods -n archtics-uc
- Grant namespaced permissions to that service account:
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: Role
  metadata:
    namespace: archtics-uc
    name: list-pods
  rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get","list","watch"]
  ---
  apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: RoleBinding
  metadata:
    name: list-pods-binding
    namespace: archtics-uc
  subjects:
  - kind: ServiceAccount
    name: default
    namespace: archtics-uc
  roleRef:
    kind: Role
    name: list-pods
    apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
2) If you truly need cluster-wide access (e.g.,
-A
) - Grant cluster-wide RBAC to the service account:
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: ClusterRole
  metadata:
    name: list-pods-clusterwide
  rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get","list","watch"]
  ---
  apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: ClusterRoleBinding
  metadata:
    name: list-pods-clusterwide-binding
  subjects:
  - kind: ServiceAccount
    name: default
    namespace: archtics-uc
  roleRef:
    kind: ClusterRole
    name: list-pods-clusterwide
    apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
Prefect-specific notes - If this is running in a Prefect Kubernetes work pool, your flow runs inside a pod that uses a service account. If you didn’t set one explicitly, it’s usually the namespace’s
default
SA (as shown in your error). You can: - Create a dedicated service account with the needed permissions (recommended), and - Set
serviceAccountName
in your work pool’s base job template or job/pod spec so Prefect jobs use that SA. - Alternatively, narrow the kubectl commands to your namespace and grant only namespaced permissions. Helpful checks - See what the SA can do:
Copy code
kubectl auth can-i list pods --as=system:serviceaccount:archtics-uc:default -n archtics-uc
  kubectl auth can-i list pods --as=system:serviceaccount:archtics-uc:default -A
References - Prefect Kubernetes work pool guide: Kubernetes work pool guide - Kubernetes RBAC overview: K8s RBAC docs - Checking access: kubectl auth can-i If you can share how you’re running the command (shell task in a flow vs. something in a deployment step) and which work pool you’re using, I can suggest the exact place to set
serviceAccountName
or adjust the command.
c
@Marvin I updated my script to just look at the archtics-uc namespace and received the following error:
Copy code
PID 33 stream output:
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:archtics-uc:default" cannot list resource "pods" in API group "" in the namespace "archtics-uc"
m
thought for 268.5 seconds
You’re still hitting a Kubernetes RBAC denial. Even within the archtics-uc namespace, the pod is using the namespace’s default service account: systemserviceaccountarchtics-uc:default and that account does not have permission to list Pods in that namespace. How to fix Pick one of these approaches: A) Grant the default SA the minimal rights in archtics-uc - Apply a Role and RoleBinding:
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: Role
  metadata:
    namespace: archtics-uc
    name: list-pods
  rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get","list","watch"]
  ---
  apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: RoleBinding
  metadata:
    name: list-pods-binding
    namespace: archtics-uc
  subjects:
  - kind: ServiceAccount
    name: default
    namespace: archtics-uc
  roleRef:
    kind: Role
    name: list-pods
    apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
B) Recommended: Create a dedicated SA and bind it - Use a separate service account and bind only what you need:
Copy code
apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: archtics-uc-prefect
    namespace: archtics-uc
  ---
  apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: Role
  metadata:
    namespace: archtics-uc
    name: list-pods
  rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get","list","watch"]
  ---
  apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: RoleBinding
  metadata:
    name: list-pods-binding
    namespace: archtics-uc
  subjects:
  - kind: ServiceAccount
    name: archtics-uc-prefect
    namespace: archtics-uc
  roleRef:
    kind: Role
    name: list-pods
    apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
Prefect-specific step (Kubernetes work pool) - Make sure your Prefect jobs use that service account. In a Kubernetes work pool, set
serviceAccountName
in the pod spec of the base job template (path:
spec.template.spec.serviceAccountName
in the Job manifest). Then update the work pool to use that template:
Copy code
prefect work-pool update <your-pool-name> --base-job-template path/to/template.json
Docs: Kubernetes work pool guide, Manage work pools Validate permissions - Before re-running the flow, check access:
Copy code
kubectl auth can-i list pods -n archtics-uc --as=system:serviceaccount:archtics-uc:default
  kubectl auth can-i list pods -n archtics-uc --as=system:serviceaccount:archtics-uc:archtics-uc-prefect
References - Kubernetes RBAC overview: K8s RBAC docs - Checking API access: kubectl auth can-i If you share your work pool name and whether you’d like to use a dedicated SA, I can sketch the exact base job template snippet to set
serviceAccountName
in your pool.