i can share complete kube runner yaml file if required

we were upgrading from prefect 2 from prefect 1

we have runner yaml file with configuration and deploying using Kustomize

I execute below command, kind is set to statefulset instead of deployment when i look at UI the namespace is set to default

kubectl describe statefulset runner -n maestro

Output:

Name:               runner
Namespace:          maestro
CreationTimestamp:  Mon, 08 Jul 2024 13:10:49 -0600
Selector:           app=maestro,process=runner
Labels:             app=maestro
                    <http://app.kubernetes.io/instance=maestro|app.kubernetes.io/instance=maestro>
                    process=runner
Annotations:        <none>
Replicas:           1 desired | 1 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           <http://admission.datadoghq.com/enabled=true|admission.datadoghq.com/enabled=true>
                    app=maestro
                    envoy-flavor=<http://xxxxx.com/patch-ndots=true|xxxxx.com/patch-ndots=true>
                    process=runner
                    <http://tags.xxxxxxxx.com/service=maestro|tags.xxxxxxxx.com/service=maestro>
                    <http://tags.xxxxxxxx.com/version=52e9b76806e0152534af0a9d99c457d864a86d5f|tags.xxxxxxxx.com/version=52e9b76806e0152534af0a9d99c457d864a86d5f>
                    version=stable
  Annotations:      co.elastic.logs/enabled: true
                    <http://proxy.istio.io/config|proxy.istio.io/config>: {"holdApplicationUntilProxyStarts": true}
  Service Account:  maestro
  Containers:
   maestro:
    Image:      docker-local.artifactory.internal/maestro-reprise:52e9b76806e0152534af0a9d99c457d864a86d5f
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/bash
      -c
    Args:
      prefect config set PREFECT_API_URL=$PREFECT_API_URL
      prefect config set PREFECT_API_KEY=$PREFECT_API_KEY
      prefect deploy --all
      prefect worker start --pool omnibus-data-ops-pool
      
    Limits:
      cpu:     4
      memory:  3Gi
    Requests:
      cpu:     1
      memory:  2Gi
    Environment Variables from:
      env-4tg25mhd4b  ConfigMap  Optional: false
      env-542m7b2gm6  Secret     Optional: false
    Environment:      <none>
    Mounts:
      /srv/prefect/tmp from maestro-runner-persistent-storage (rw)
  Volumes:  <none>
Volume Claims:
  Name:          maestro-runner-persistent-storage
  StorageClass:  xfs-rook-block
  Labels:        <none>
  Annotations:   <none>
  Capacity:      50Gi
  Access Modes:  [ReadWriteOnce]
Events:          <none>

@Kevin Grismore Please let me know if you need any additional information. Thanks

I think my primary suggestion at the moment is refer to the helm chart for the role, role binding, and service account that should be attached to the worker. https://github.com/PrefectHQ/prefect-helm/tree/main/charts/prefect-worker

even if you don't use it directly, you should probably try imitating all the components it represents

you can see in the role which permissions on each api group the worker needs to do its job successfully

@Kevin Grismore, Still facing connection issue. please find yaml files which i have created and attached base job template, is there any misconfiguration you can point out or missing any. Thanks role.yaml

apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: runner
  namespace: maestro
rules:
- apiGroups: [""]
  resources: ["events", "pods", "pods/log", "pods/status"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

role-binding.yaml

apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: runner
  namespace: maestro
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: Role
  name: runner
subjects:
- kind: ServiceAccount
  name: maestro
  namespace: maestro

service_account.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: runner
  namespace: maestro

deployment_runner.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: runner
  labels:
    app: maestro
    process: runner
  annotations:
spec:
  replicas: 0
  selector:
    matchLabels:
      app: maestro
      process: runner
  template:
    metadata:
      labels:
        <http://admission.datadoghq.com/enabled|admission.datadoghq.com/enabled>: "true"
        app: maestro
        envoy-flavor: mx
        process: runner
        version: stable
        <http://mx.com/patch-ndots|mx.com/patch-ndots>: "true"
      annotations:
        co.elastic.logs/enabled: "true"
        <http://proxy.istio.io/config|proxy.istio.io/config>: '{"holdApplicationUntilProxyStarts": true}'
    spec:
      serviceAccountName: maestro
      containers:
        - image: docker/maestro-reprise
          name: maestro
          resources:
            requests:
              memory: 2Gi #TODO
              cpu: 1000m #TODO
            limits:
              memory: 3Gi #TODO
              cpu: 4000m #TODO
          command: ["/bin/bash", "-c"]
          args:
            - |
              prefect config set PREFECT_API_URL=$PREFECT_API_URL
              prefect config set PREFECT_API_KEY=$PREFECT_API_KEY
              prefect deploy --all
              prefect worker start --pool omnibus-data-ops-pool
          readinessProbe:
            #TODO
          env:
          envFrom:
            - configMapRef:
                name: env
            - secretRef:
                name: env
          volumeMounts:
            - name: maestro-runner-persistent-storage
              mountPath: /srv/prefect/tmp
  volumeClaimTemplates:
    - metadata:
        name: maestro-runner-persistent-storage
      spec:
        storageClassName: xfs-rook-block
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 50Gi

base_job_template

WorkPool(
    id='8d959567-796f-437e-b0fa-aef3d8f13e49',
    created=DateTime(2024, 7, 3, 20, 28, 7, 22301, tzinfo=FixedTimezone(0, name="+00:00")),
    updated=DateTime(2024, 7, 12, 18, 36, 15, 78634, tzinfo=FixedTimezone(0, name="+00:00")),
    name='omnibus-data-ops-pool',
    type='kubernetes',
    base_job_template={
        'variables': {
            'type': 'object',
            'properties': {
                'env': {
                    'type': 'object',
                    'title': 'Environment Variables',
                    'description': 'Environment variables to set when starting a flow run.',
                    'additionalProperties': {'type': 'string'}
                },
                'name': {'type': 'string', 'title': 'Name', 'default': 'runner', 'description': 'Name given to infrastructure created by a worker.'},
                'image': {
                    'type': 'string',
                    'title': 'Image',
                    'example': '<http://docker.io/prefecthq/prefect:2-latest|docker.io/prefecthq/prefect:2-latest>',
                    'description': 'The image reference of a container image to use for created jobs. If not set, the latest Prefect image will be used.'
                },
                'labels': {'type': 'object', 'title': 'Labels', 'description': 'Labels applied to infrastructure created by a worker.', 'additionalProperties': {'type': 'string'}},
                'command': {
                    'type': 'string',
                    'title': 'Command',
                    'description': 'The command to use when starting a flow run. In most cases, this should be left blank and the command will be automatically generated by the worker.'
                },
                'namespace': {'type': 'string', 'title': 'Namespace', 'default': 'maestro', 'description': 'The Kubernetes namespace to create jobs within.'},
                'stream_output': {'type': 'boolean', 'title': 'Stream Output', 'default': True, 'description': 'If set, output will be streamed from the job to local standard output.'},
                'cluster_config': {
                    'allOf': [{'$ref': '#/definitions/KubernetesClusterConfig'}],
                    'title': 'Cluster Config',
                    'description': 'The Kubernetes cluster config to use for job creation.'
                },
                'finished_job_ttl': {
                    'type': 'integer',
                    'title': 'Finished Job TTL',
                    'description': 'The number of seconds to retain jobs after completion. If set, finished jobs will be cleaned up by Kubernetes after the given delay. If not set, jobs will
be retained indefinitely.'
                },
                'image_pull_policy': {
                    'enum': ['IfNotPresent', 'Always', 'Never'],
                    'type': 'string',
                    'title': 'Image Pull Policy',
                    'default': 'IfNotPresent',
                    'description': 'The Kubernetes image pull policy to use for job containers.'
                },
                'service_account_name': {'type': 'string', 'title': 'Service Account Name', 'description': 'The Kubernetes service account to use for job creation.'},
                'job_watch_timeout_seconds': {
                    'type': 'integer',
                    'title': 'Job Watch Timeout Seconds',
                    'description': 'Number of seconds to wait for each event emitted by a job before timing out. If not set, the worker will wait for each event indefinitely.'
                },
                'pod_watch_timeout_seconds': {
                    'type': 'integer',
                    'title': 'Pod Watch Timeout Seconds',
                    'default': 60,
                    'description': 'Number of seconds to watch for pod creation before timing out.'
                }
            },
            'definitions': {
                'KubernetesClusterConfig': {
                    'type': 'object',
                    'title': 'KubernetesClusterConfig',
                    'required': ['config', 'context_name'],
                    'properties': {
                        'config': {'type': 'object', 'title': 'Config', 'description': 'The entire contents of a kubectl config file.'},
                        'context_name': {'type': 'string', 'title': 'Context Name', 'description': 'The name of the kubectl context to use.'}
                    },
                    'description': 'Stores configuration for interaction with Kubernetes clusters.\n\nSee `from_file` for creation.',
                    'secret_fields': [],
                    'block_type_slug': 'kubernetes-cluster-config',
                    'block_schema_references': {}
                }
            },
            'description': 'Default variables for the Kubernetes worker.\n\nThe schema for this class is used to populate the `variables` section of the default\nbase job template.'
        },
        'job_configuration': {
            'env': '{{ env }}',
            'name': '{{ name }}',
            'labels': '{{ labels }}',
            'command': '{{ command }}',
            'namespace': '{{ namespace }}',
            'job_manifest': {
                'kind': 'Job',
                'spec': {
                    'template': {
                        'spec': {
                            'containers': [{'env': '{{ env }}', 'args': '{{ command }}', 'name': 'prefect-job', 'image': '{{ image }}', 'imagePullPolicy': '{{ image_pull_policy }}'}],
                            'completions': 1,
                            'parallelism': 1,
                            'restartPolicy': 'Never',
                            'serviceAccountName': '{{ service_account_name }}'
                        }
                    },
                    'backoffLimit': 0,
                    'ttlSecondsAfterFinished': '{{ finished_job_ttl }}'
                },
                'metadata': {'labels': '{{ labels }}', 'namespace': '{{ namespace }}', 'generateName': '{{ name }}-'},
                'apiVersion': 'batch/v1'
            },
            'stream_output': '{{ stream_output }}',
            'cluster_config': '{{ cluster_config }}',
            'job_watch_timeout_seconds': '{{ job_watch_timeout_seconds }}',
            'pod_watch_timeout_seconds': '{{ pod_watch_timeout_seconds }}'
        }
    },
    concurrency_limit=5,
    status=WorkPoolStatus.READY,
    default_queue_id='70e699f3-4483-48be-aa3b-e2c5a4bb2573'
)

prefect.yaml

definitions:
    work_pools:
        work_pool: &work_pool
            name: test-omnibus-data-ops-pool

deployments:

- name: test_flows
  entrypoint: flows/test.py:run_test
  schedules:
    - cron: "15 * * * *" # At 15 minutes past the hour, every hour, every day
  work_pool: *work_pool

Still same issue

kubectl get role runner -n maestro -o yaml

output:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"Role","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"maestro"},"name":"runner","namespace":"maestro"},"rules":[{"apiGroups":[""],"resources":["events","pods","pods/log","pods/status"],"verbs":["get","watch","list"]},{"apiGroups":["batch"],"resources":["jobs"],"verbs":["get","list","watch","create","update","patch","delete"]}]}
  creationTimestamp: "2024-07-12T15:13:39Z"
  labels:
    app.kubernetes.io/instance: maestro
  name: runner
  namespace: maestro
  resourceVersion: "1167285778"
  uid: 06d5ef39-1dc0-4a19-9773-fbd8e2a1e916
rules:
- apiGroups:
  - ""
  resources:
  - events
  - pods
  - pods/log
  - pods/status
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete

kubectl get serviceaccount maestro -n maestro -o yaml Output:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"labels":{"account":"maestro","app.kubernetes.io/instance":"maestro","datadog":"consumer","dmzegress":"consumer","dmzproxy":"consumer","netpolset":"app-default","pgproxy":"consumer","provider":"maestro","vertica":"consumer"},"name":"maestro","namespace":"maestro"}}
  creationTimestamp: "2024-06-25T20:40:55Z"
  labels:
    account: maestro
    app.kubernetes.io/instance: maestro
    datadog: consumer
    dmzegress: consumer
    dmzproxy: consumer
    netpolset: app-default
    pgproxy: consumer
    provider: maestro
    vertica: consumer
  name: maestro
  namespace: maestro
  resourceVersion: "1148806698"
  uid: 13fdad24-a527-4373-99ab-45a28032e630

kubectl get rolebindings -n maestro -o yaml output:

apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"RoleBinding","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"namespaces","sand-security":"open"},"name":"application-core:teleport-application-role","namespace":"maestro"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"teleport-application-role"},"subjects":[{"apiGroup":"rbac.authorization.k8s.io","kind":"Group","name":"application-core"},{"apiGroup":"rbac.authorization.k8s.io","kind":"Group","name":"system:authenticated"}]}
    creationTimestamp: "2024-06-25T20:41:17Z"
    labels:
      app.kubernetes.io/instance: namespaces
      sand-security: open
    name: application-core:teleport-application-role
    namespace: maestro
    resourceVersion: "1146528188"
    uid: da570fc7-9966-48e1-a5f5-38646dafdc7a
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: teleport-application-role
  subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: application-core
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: system:authenticated
- apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"RoleBinding","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"maestro"},"name":"runner","namespace":"maestro"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"Role","name":"runner"},"subjects":[{"kind":"ServiceAccount","name":"maestro","namespace":"maestro"}]}
    creationTimestamp: "2024-07-12T15:13:39Z"
    labels:
      app.kubernetes.io/instance: maestro
    name: runner
    namespace: maestro
    resourceVersion: "1167594198"
    uid: f658589d-b219-4b79-80e7-61b284745353
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: runner
  subjects:
  - kind: ServiceAccount
    name: maestro
    namespace: maestro
kind: List
metadata:
  resourceVersion: ""

I analyzed below things not sure what im missing 1. role "runner": ◦ has appropriate permissions for pods and jobs ◦ allows creating, updating, and deleting jobs 2. serviceAccount "maestro": ◦ exists in the "maestro" namespace ◦ has various labels attached 3. rolebinding "runner": ◦ correctly binds the "runner" Role to the "maestro" ServiceAccount ◦ is in the correct namespace (maestro) ◦ the RBAC (Role-Based Access Control) setup appears to be correct and the "maestro" servicecccount should have the necessary permissions to create and manage jobs in the "maestro" namespace. @Kevin Grismore can you please point out what im missing here. Appreciate your help on this. Thanks!!

below is the egress permission we have added, do you think we need to add ingress as well

metadata:
  name: egress-routes
spec:
  routes:
    - host: api.prefect.cloud
      port: 443

tried all the role, role binding permission to service account, not sure where we are going wrong