Hey all! I am using the latest Prefect version (2.6.3). The agent is deployed on AKS in the namespace called

dev-prefect-2

. Which makes no sense - why would I need privileges in the

default

namespace? I can run this with Prefect 2.3 without issues.

Hi Klemen, would you mind moving your logs into the reply section so it doesn’t take up the full screen of the chat for other users?

Deployment yaml:

###
### A complete description of a Prefect Deployment for flow 'healthcheck'
###
name: dev-prefect-2
description: null
version: b0839155eb142bba0846b25c771ea2f8
# The work queue that will handle this deployment's runs
work_queue_name: kubernetes
tags:
- kubernetes
parameters: {}
schedule: null
infra_overrides: {}

###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: healthcheck
manifest_path: null
infrastructure:
  type: kubernetes-job
  env: {}
  labels: {}
  name: null
  command: null
  image: <http://aiknauf.azurecr.io/prefect-azure-3.10:latest|aiknauf.azurecr.io/prefect-azure-3.10:latest>
  namespace: dev-prefect-2
  service_account_name: null
  image_pull_policy: null
  cluster_config: null
  job:
    apiVersion: batch/v1
    kind: Job
    metadata:
      labels: {}
    spec:
      template:
        spec:
          parallelism: 1
          completions: 1
          restartPolicy: Never
          containers:
          - name: prefect-job
            env: []
  customizations: []
  job_watch_timeout_seconds: 5
  pod_watch_timeout_seconds: 60
  stream_output: true
  finished_job_ttl: null
  _block_document_id: c4d68193-e5b3-4a5e-b9fd-c03920fd9cc9
  _block_document_name: dev-k8s-job
  _is_anonymous: false
  block_type_slug: kubernetes-job
  _block_type_slug: kubernetes-job
storage:
  bucket_path: prefect2/dev
  azure_storage_connection_string: null
  azure_storage_account_name: '**********'
  azure_storage_account_key: '**********'
  _block_document_id: 1cefdc9c-5949-4477-98ff-59561fc388c3
  _block_document_name: dev-azure-storage
  _is_anonymous: false
  block_type_slug: azure
  _block_type_slug: azure
path: ''
entrypoint: healthcheck.py:healthcheck
parameter_openapi_schema:
  title: Parameters
  type: object
  properties: {}
  required: null
  definitions: null

Agent deployment yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-agent
  namespace: dev-prefect-2
  labels:
    app: prefect-agent
spec:
  selector:
    matchLabels:
      app: prefect-agent
  replicas: 1
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - name: agent
        image: prefecthq/prefect:2.6.3-python3.10
        command: ["prefect", "agent", "start", "-q", "kubernetes"]
        imagePullPolicy: "IfNotPresent"
        env:
          - name: PREFECT_API_URL
            value: <https://api.prefect.cloud/api/accounts/f0e45360-d22b-40ea-a87d-6129178b8311/workspaces/af0d1913-8b92-46ed-9ccb-f5cd0915b1fb>
          - name: PREFECT_API_KEY
            value: "*****""
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  namespace: dev-prefect-2
  name: prefect-agent
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/status"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-agent-role-binding
  namespace: dev-prefect-2
subjects:
- kind: ServiceAccount
  name: dev-prefect-2
  namespace: dev-prefect-2
roleRef:
  kind: Role
  name: prefect-agent
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>

when I run the flow the agent logs this:

kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd0563ae3-fe14-4f5f-a96b-3f691f716e11', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'd51df1b6-cbd8-4c4e-8d59-5dd5621cbca2', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f1348ce8-bbb0-4a7a-ba1c-5653cead2c0b', 'Date': 'Thu, 20 Oct 2022 11:56:52 GMT', 'Content-Length': '317'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:dev-prefect-2:default\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}