Short answer: there isn’t a special “enable tasks” setting for the Kubernetes work pool worker. When the UI shows only flow logs and no graph, it’s almost always because the flow body is doing the work directly without any

@task

functions (or subflows) being called. Prefect only draws the graph and shows task/subflow details when your code uses

@task

and

@flow

-decorated callables. What to check - Are you decorating the units of work with

@task

and calling them from a

@flow

? - If everything happens inside the flow body (plain Python), you’ll see flow logs but no task/subflow nodes in the UI. - If you intend to show subflows, make sure the nested function is decorated with

@flow

and is called from within your parent flow. - Version consistency: ensure your image used by the job has a recent Prefect 3.x version compatible with your server/Cloud workspace. Minimal example that will show up in the UI graph

from prefect import flow, task

@task
def do_work(x: int) -> int:
    return x * 2

@flow
def parent():
    # This creates a task run (shows in graph)
    y = do_work(21)
    return y

if __name__ == "__main__":
    parent()

Kubernetes worker-specific notes - If your worker is successfully creating jobs and you see flow logs in the UI, the required API settings for orchestration are already in place. - If you’re using a heavily customized Kubernetes job template, confirm that the job pod still has Prefect’s API environment variables set (the worker injects these by default). You can verify inside a running flow pod:

kubectl exec -it <flow-pod-name> -- env | grep PREFECT_API

You should see

PREFECT_API_URL

and

PREFECT_API_KEY

. If the pod can fetch work and send logs, these are usually fine. If you’d like, paste a small snippet of your flow showing how you’re doing the work, and (if applicable) your custom job template. I can confirm whether tasks/subflows are being created as expected. Helpful docs - Tasks concept: Tasks in Prefect 3 - Flows and subflows: Flows and Subflows - Kubernetes work pools: prefect-kubernetes collection

Got it — thanks for the extra context. Since the same code shows the task/subflow graph with process and Docker pools, the issue is specific to how your Kubernetes jobs are launched. Key point: with the Kubernetes work pool, logs can appear in the UI even if the flow-run engine in the pod never started, because the Kubernetes observer can forward pod logs to Prefect independently. That’s why logs alone aren’t proof that the pod executed

prefect flow-run execute

(which is required for task/subflow tracking). Most common Kubernetes-only causes of “logs but no graph” 1) Custom job template overrides the entrypoint and bypasses Prefect’s engine - If your template sets `command`/`args` without including

{{ command }}

, the pod will not run

prefect flow-run execute

, so no task/subflow graph is reported. - Fix: Ensure your container actually executes Prefect’s runner. Either: - Include the placeholder:

args: ["{{ command }}"]

- Or leave

args

unset and let the worker populate it. - Also be mindful of container ENTRYPOINTs: if your image has a custom ENTRYPOINT, Kubernetes

args

become parameters to that ENTRYPOINT. Ensure the final executed command is the Prefect CLI runner, not your app entrypoint. 2) Image/CLI mismatch - The container image must include a compatible Prefect 3.x installation and the

prefect

CLI. If

prefect flow-run execute

isn’t available (or fails early), the observer can still forward logs, but no tasks will be reported. 3) Pod never reaches the runner - If the pod crashes during startup/imports, tasks won’t be created. The observer still forwards logs. Quick checks to pinpoint the issue - Inspect the actual container args/command on a created pod:

kubectl get pod <pod-name> -o jsonpath='{.spec.containers[0].command}{" "}{.spec.containers[0].args}'

You should see something equivalent to

prefect flow-run execute ...

. If not, your template or image ENTRYPOINT is overriding it. - Look for the runner starting in pod logs: - You should see lines like “Executing flow run …” from Prefect’s runner. If you never see these, the engine likely didn’t start. - Verify Prefect is present in the image:

kubectl exec -it <pod-name> -- prefect version

- If you have a custom job template, make sure it includes the Prefect command placeholder:

spec:
    template:
      spec:
        restartPolicy: Never
        serviceAccountName: "{{ service_account_name }}"
        containers:
        - name: flow
          image: "{{ image }}"
          args: ["{{ command }}"]    # <- critical
          env: "{{ env }}"
          resources: "{{ resources }}"

If your image has a strict ENTRYPOINT, explicitly set

command

to a shell and pass the placeholder in

args

, e.g.:

command: ["/bin/sh", "-c"]
  args: ["{{ command }}"]

If you share: - Your work pool’s custom job template (if any) - The container image/tag you’re using - A snippet of the first 50 lines of the flow pod logs …I can confirm exactly where it’s getting bypassed. Helpful docs - Kubernetes integration (work pool, templates, observer): prefect-kubernetes docs

Thanks for confirming the flow is actually running in the pod. Given that: - The same code shows a full task/subflow graph in process/Docker pools, and - Only the Kubernetes pool is missing the graph while still showing logs, the remaining likely causes are about task-run event emission/recording in the Kubernetes run environment (not the code and not just the entrypoint). Why logs can show without a task graph on Kubernetes - The Kubernetes observer can forward pod logs to the Prefect UI independently of the flow process’ orchestration events. So seeing logs does not guarantee the pod is emitting task/subflow state events to the API. Most common Kubernetes-only causes and how to check 1) Client-side orchestration telemetry disabled in the pod - If

PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY=false

is set in the pod env, the flow engine will not emit task/subflow state events. You’ll get logs (via observer) but no task graph. - Check inside a running flow pod:

kubectl exec -it <pod> -- env | sort | grep PREFECT_

Ensure you do NOT see

PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY=false

. 2) Hitting a different API (Cloud vs self-hosted server) for Kubernetes runs - If your K8s work pool (or job template) points to a different API URL than your process/Docker pools (e.g., an internal Prefect Server), task recording may be disabled there while it works elsewhere. - Confirm the pod’s API target matches what your working pools use:

kubectl exec -it <pod> -- env | grep -E "PREFECT_API_URL|PREFECT_API_KEY"

Compare with the env used by the process/Docker pools. 3) Server-side “Task Run Recorder” service disabled (only for self-hosted Prefect Server) - On Prefect Server (OSS), task visibility depends on the Task Run Recorder service. If it’s disabled, you’ll see flow runs/logs but no tasks. - On the server, check: - Env/config flag:

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED

should be true - Or via config:

prefect config view | grep task_run_recorder

- This would explain why only runs hitting that server (your K8s pods) lack task graphs. 4) Prefect version mismatch in the image - If the Kubernetes image has an older or incompatible Prefect version, task-run event emission can break while the flow still runs and logs appear via the observer. - Check:

kubectl exec -it <pod> -- prefect version

Compare with the version used by your working pools. Targeted next steps - From a recent K8s flow pod: 1) Print Prefect env

kubectl exec -it <pod> -- env | sort | grep PREFECT_

Ensure: - `PREFECT_API_URL`/`PREFECT_API_KEY` are present and point to the same workspace as your working pools -

PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY

is not set to false 2) Print Prefect version

kubectl exec -it <pod> -- prefect version

3) If you’re on self-hosted Prefect Server for K8s runs, verify the Task Run Recorder service is enabled on that API instance. If you can share: - Whether these K8s runs are targeting Prefect Cloud or a self-hosted Prefect Server - The output of the two commands above (mask secrets) - Any work pool job template overrides that modify env injection …I can pinpoint which of the above is in play and help you fix it quickly. References - Kubernetes integration (work pools, templates, observer): prefect-kubernetes docs - Configuration guide: Prefect configuration

Sorry, I encountered an error while processing your request. Please try again.

Short answer: your other pools are almost certainly pointing at a different API backend (e.g., Prefect Cloud) where the Task Run Recorder is enabled. The Kubernetes jobs are hitting your self‑hosted Prefect Server where

task_run_recorder.enabled=false

, so tasks/subflows aren’t persisted—hence no graph—while logs still show because the Event Persister service is enabled and logs are stored as events. Why this happens - In Prefect 3.x, task/subflow visibility relies on the server-side Task Run Recorder. If it’s disabled, the UI won’t show task runs or graphs. - Logs are persisted by the Event Persister service, which can be enabled independently—so logs can appear even when task graphs don’t. - Process and Docker pools typically inherit your local Prefect Cloud login/config, so they may be sending runs to Cloud (recorder enabled) while your Kubernetes pool is configured (via work pool/env) to send runs to your self-hosted server (recorder disabled). How to confirm - Check the API target in each environment:

# Local (process/Docker) environment
  prefect config view | grep -E "api\.url|cloud\.api"
  env | grep -E "PREFECT_API_URL|PREFECT_API_KEY"

  # Inside a Kubernetes flow pod
  kubectl exec -it <pod> -- env | grep -E "PREFECT_API_URL|PREFECT_API_KEY"

You’ll likely find process/Docker use Cloud, and the K8s pods use your self-hosted API. How to fix (pick one) - Re-enable the Task Run Recorder on your Prefect Server and restart services:

export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true
  # or via config file: server.services.task_run_recorder.enabled=true

- Or point the Kubernetes work pool/pods to the same API as your other pools (e.g., Prefect Cloud) via work pool variables/env. Helpful docs - Task Run Recorder (what it is and the setting): Server service: Task Run Recorder - Task Run Recorder service API reference: Task Run Recorder API - Self-hosted architecture/services: Self-hosted architecture

Short answer: No —

PREFECT_RESULTS_PERSIST_LOCATION

has nothing to do with task/subflow graphs. It only controls where your task/flow return values are stored. Task/subflow visibility in the UI comes from task-run events being recorded by your server’s Task Run Recorder service. Given your details, here’s what explains the behavior and how to verify it: Why logs show up but the task graph doesn’t - Logs can come from the Kubernetes Observer, which forwards pod logs to your server independently of the flow process’ task instrumentation. So “logs visible” does not mean task events were emitted or recorded. - The task/subflow graph appears only when: 1) The flow process emits task state events (client-side), and 2) The server’s Task Run Recorder service is enabled and consuming those events into TaskRun records. You mentioned

task_run_recorder.enabled: false

on your server. If that flag is actually applied to the API instance receiving the events, new task runs will not be materialized — so no graph — even though logs still arrive. But other pools show graphs — how? - Most commonly, they’re hitting a different API instance (or replica) than your Kubernetes jobs. In Kubernetes, it’s easy for different callers to resolve/load-balance to different pods with different env/config. One API pod may have the recorder enabled while another has it disabled. - Another possibility: the setting you saw is not the one applied to the running API process (e.g., you checked a client machine’s config or an old Deployment, not the live API pod). - Less common: your K8s flow pods may have

PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY=false

, which stops the client from emitting task events at all. In that case you’d see logs (observer) but no tasks, while your other pools (without that flag) still show tasks. What to check next (targeted) 1) Confirm the setting on the actual API pod(s) that your runs hit - Check env on each API replica:

kubectl exec -it -n <ns> <prefect-api-pod> -- env | grep PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED
  kubectl exec -it -n <ns> <prefect-api-pod> -- env | grep PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED

Ensure the recorder is enabled (true) on all API pods behind your service. - If possible, also check the service logs for the recorder starting:

kubectl logs -n <ns> <prefect-api-pod> | grep -i "Task Run Recorder"

2) Verify the flow pod is emitting task events (not disabled by telemetry flag) - In a recent flow pod from the Kubernetes pool:

kubectl exec -it <flow-pod> -- env | sort | grep PREFECT_

Ensure you do NOT see

PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY=false

. 3) Sanity-check Prefect versions - Make sure the image used by the Kubernetes jobs has a compatible Prefect 3.x and the

prefect

CLI:

kubectl exec -it <flow-pod> -- prefect version

4) Ensure all pools truly use the same API URL - Compare what each environment is using (process/Docker vs K8s): - Local (process/Docker):

prefect config view | grep -E "api\.url|cloud\.api"
    env | grep -E "PREFECT_API_URL|PREFECT_API_KEY"

- Inside a K8s flow pod:

kubectl exec -it <flow-pod> -- env | grep -E "PREFECT_API_URL|PREFECT_API_KEY"

What will actually resolve the issue - Enable the Task Run Recorder on the server instance(s) your K8s pods hit:

# Server env
  PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true
  # Keep the Event Persister on:
  PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true

Then restart the API deployment so the setting applies uniformly to all replicas. - If you find

PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY=false

set in the K8s job pods, remove it so the client emits task events. References - Task Run Recorder setting: Server service: Task Run Recorder

- Task Run Recorder API reference: Task Run Recorder service - Self-hosted architecture/services: Self-hosted architecture Bottom line: you do not need

PREFECT_RESULTS_PERSIST_LOCATION

for UI graphs. Align the server config so Task Run Recorder is enabled on the API instance(s) your Kubernetes jobs use, and ensure the flow pods aren’t disabling client telemetry. That will restore task/subflow visibility.

Great question — and you’re on a recent Prefect 3.x, so you’ve got built-in OpenTelemetry hooks available. Below are concise, verified steps to enable OTEL for the Prefect API server and Prefect Server “services” separately on 3.6.22, plus the packages you’ll need. Assumptions - You’re on Prefect 3.6.22. - You have an OTLP-compatible collector (e.g., Jaeger, Tempo) available. Required Python packages - Recommended (installs exporter + auto-instrumentation):

pip install "prefect[otel]"

- Or install individually (equivalent/minimal):

pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-instrumentation

- Optional (better server traces if you want framework/db spans):

pip install opentelemetry-instrumentation-fastapi opentelemetry-instrumentation-sqlalchemy opentelemetry-instrumentation-logging

Common OTEL environment Set these before starting any process (server or services):

export OTEL_SERVICE_NAME=prefect-server        # or prefect-services when starting services
export OTEL_TRACES_EXPORTER=otlp
# Choose one protocol + endpoint that matches your collector:
# gRPC (default)
export OTEL_EXPORTER_OTLP_ENDPOINT=<http://localhost:4317>
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
# or HTTP
# export OTEL_EXPORTER_OTLP_ENDPOINT=<http://localhost:4318>
# export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

# Optional:
# export OTEL_TRACES_SAMPLER=parentbased_traceidratio
# export OTEL_TRACES_SAMPLER_ARG=0.2
# export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=<API_KEY>"   # if your vendor requires headers

Enable OTEL for the Prefect API server ONLY - Run the webserver without services and wrap it with the OTEL auto-instrumenter:

opentelemetry-instrument prefect server start --no-services

- Tip: If you want to rename the service to be explicit for the API, set

OTEL_SERVICE_NAME=prefect-api

. Enable OTEL for Prefect Server “services” ONLY - Start just the background services (scheduler, late runs, etc.) in a separate process:

# Optionally control which services run:
# export PREFECT_API_SERVICES_SCHEDULER_ENABLED=true
# export PREFECT_API_SERVICES_LATE_RUNS_ENABLED=true

export OTEL_SERVICE_NAME=prefect-services
opentelemetry-instrument prefect server services start

Run both with OTEL together (one process) - If you prefer a single process for both webserver + services:

export OTEL_SERVICE_NAME=prefect-server
opentelemetry-instrument prefect server start

Notes and tips - Framework/db spans: If you install

opentelemetry-instrumentation-fastapi

and

opentelemetry-instrumentation-sqlalchemy

, the

opentelemetry-instrument

wrapper will pick them up and emit richer spans for API routes and DB calls. - Flow/task spans vs server/services spans: - Server/services OTEL above is “process auto-instrumentation” (FastAPI/SQLAlchemy/etc). - Prefect also emits orchestration traces for flow/task runs when enabled. That’s controlled by:

export PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY=true   # default is true

- If you also want OS-level CPU/memory metrics during flow execution (not the server), enable:

export PREFECT_TELEMETRY_ENABLE_RESOURCE_METRICS=true       # default is true
    # And point your metrics exporter to OTLP HTTP if needed:
    # export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=<http://localhost:4318/v1/metrics>

- Verified CLI flags: -

prefect server start --no-services

runs just the API webserver. -

prefect server services start

runs the background services. - Quick Jaeger test collector:

docker run -d -p 4317:4317 -p 4318:4318 -p 16686:16686 jaegertracing/all-in-one:latest
  # Then browse <http://localhost:16686> to see traces

Docs and references - Telemetry concepts: Telemetry concepts - Telemetry settings API: Telemetry settings (API ref)

If you share which collector you’re targeting (Jaeger, Tempo, Honeycomb, etc.), I can tailor the exact env var examples (e.g., headers, protocol) for that backend.

Short answer: put the sidecar (and any required volumes/volumeMounts/initContainers) directly into your Kubernetes work pool’s base job template under spec.template.spec. Prefect’s Kubernetes worker will keep using the first container (prefect-job) to run your flow, and it will leave your additional containers untouched. Here’s a complete base job template example that adds a Kerberos sidecar which performs kinit and periodically renews the ticket in a shared cache volume. You can drop this JSON into your work pool’s base job template. Adjust the Secret/ConfigMap names to match what exists in your cluster. ``` { "variables": { "type": "object", "properties": { "namespace": { "type": "string", "title": "Namespace", "default": "default" }, "service_account_name": { "type": "string", "title": "Service Account Name", "default": null }, "image_pull_policy": { "enum": ["IfNotPresent", "Always", "Never"], "title": "Image Pull Policy", "default": "IfNotPresent" }, "backoff_limit": { "type": "integer", "title": "Backoff Limit", "default": 0, "minimum": 0 }, "finished_job_ttl": { "type": "integer", "title": "Finished Job TTL", "default": null } }, "required": ["namespace"] }, "job_configuration": { "job_manifest": { "apiVersion": "batch/v1", "kind": "Job", "metadata": { "annotations": "{{ annotations }}", "labels": "{{ labels }}", "namespace": "{{ namespace }}", "generateName": "{{ name }}-" }, "spec": { "backoffLimit": "{{ backoff_limit }}", "ttlSecondsAfterFinished": "{{ finished_job_ttl }}", "template": { "metadata": {}, "spec": { "parallelism": 1, "completions": 1, "restartPolicy": "Never", "serviceAccountName": "{{ service_account_name }}", "volumes": [ { "name": "krb5-config", "configMap": { "name": "kerberos-config" // contains krb5.conf key } }, { "name": "krb5-keytab", "secret": { "secretName": "kerberos-keytab" // contains krb5.keytab key } }, { "name": "krb5-ccache", "emptyDir": {} } ], "containers": [ { "name": "prefect-job", "image": "{{ image }}", "imagePullPolicy": "{{ image_pull_policy }}", "args": "{{ command }}", // Keep env as a list; Prefect will still inject its own and any deployment-level env "env": [ { "name": "KRB5_CONFIG", "value": "/etc/krb5.conf" }, { "name": "KRB5CCNAME", "value": "/krb5/ccache" } ], "volumeMounts": [ { "name": "krb5-config", "mountPath": "/etc/krb5.conf", "subPath": "krb5.conf" }, { "name": "krb5-ccache", "mountPath": "/krb5" } ], "resources": { "limits": { "cpu": "{{ cpu_limit }}", "memory": "{{ memory_limit }}" }, "requests": { "cpu": "{{ cpu_request }}", "memory": "{{ memory_request }}" } } }, // Kerberos sidecar to acquire + renew tickets { "name": "kerberos-sidecar", "image": "your-krb5-image:tag", // must include kinit (or k5start) "env": [ { "name": "KRB5_CONFIG", "value": "/etc/krb5.conf" }, { "name": "KRB5CCNAME", "value": "/krb5/ccache" }, {

"name": "KERBEROS_PRINCIPAL", "valueFrom": { "secretKeyRef": { "name": "kerberos-credentials", "key": "principal" } } } ], "volumeMounts": [ { "name": "krb5-config", "mountPath": "/etc/krb5.conf", "subPath": "krb5.conf" }, { "name": "krb5-keytab", "mountPath": "/etc/krb5.keytab", "subPath": "krb5.keytab" }, { "name": "krb5-ccache", "mountPath": "/krb5" } ], "command": [ "sh", "-c", "kinit -kt /etc/krb5.keytab \"$KERBEROS_PRINCIPAL\" && \ while true; do kinit -R || kinit -kt /etc/krb5.keytab \"$KERBEROS_PRINCIPAL\"; sleep 3600; done" ], "resources": { "requests": { "cpu": "50m", "memory": "64Mi" }, "limits": { "cpu": "200m", "memory": "256Mi" } } } ] } } } } } }

What changed vs. the default template
- Added three volumes under spec.template.spec.volumes:
  - krb5-config (ConfigMap with krb5.conf)
  - krb5-keytab (Secret with your keytab)
  - krb5-ccache (EmptyDir to share the credential cache file)
- Added a second container in spec.template.spec.containers (the Kerberos sidecar) that:
  - mounts the keytab, krb5.conf, and shared ccache volume
  - runs kinit once then periodically renews the ticket
- Mounted krb5.conf and the shared ccache into the prefect-job container and set KRB5CCNAME so your flow can use the ticket
- Kept prefect-job as the first container; Prefect streams logs and runs the flow in this container

How to apply this to your work pool
- Via CLI

prefect work-pool create "my-k8s-pool" --type kubernetes --base-job-template ./kerberos-base-job.json # or update an existing pool prefect work-pool update "my-k8s-pool" --base-job-template ./kerberos-base-job.json ``` - Via Prefect Cloud UI: Work Pools -> your Kubernetes pool -> Edit -> Base job template Notes and gotchas - The main flow container should remain first and usually named prefect-job - You can also use an initContainer to perform an initial kinit if you prefer a one-time setup before the flow starts; the sidecar is helpful to keep tickets fresh during long runs - Prefect will still inject its own env vars and any environment variables you set on the deployment; listing env as an array in the template is fine—your entries will be preserved and Prefect-managed env vars will be merged in - Make sure your image for both containers includes Kerberos tooling (kinit or k5start), and that the ConfigMap/Secret names match your cluster resources Docs and references - Customize base job templates: Customize base job templates - Kubernetes work pools/workers: Run on Kubernetes (work pools) - Work pools concept: Work pools (concepts) - Manage work pools: Manage work pools If you share the names of your Secrets/ConfigMaps (or whether you want initContainer-only vs. always-on renewal), I can tailor the template exactly to your setup.

You’ve run into the key difference between a Deployment and a Job: most sidecar injectors/admission webhooks look at the Pod template’s metadata (spec.template.metadata), not the parent object’s metadata. In the default Prefect Kubernetes base job template, there is no spec.template.metadata section, so labels you set at Job.metadata don’t propagate to the Pod and the injector never fires. What’s missing - spec.template.metadata.labels (and usually annotations) in your work pool’s base job template. Putting labels only at Job.metadata won’t trigger pod-level mutating webhooks. How to fix it - Add spec.template.metadata with your labels/annotations to the base job template so the created Pod has the labels the injector watches (e.g., enableIdentityHelper: "true"). Minimal patch to your base job template

{
  "variables": {
    "type": "object",
    "properties": {
      "namespace": {
        "type": "string",
        "default": "default"
      },
      "annotations": {
        "type": "object",
        "default": {}
      },
      "labels": {
        "type": "object",
        "default": {}
      }
    },
    "required": [
      "namespace"
    ]
  },
  "job_configuration": {
    "job_manifest": {
      "apiVersion": "batch/v1",
      "kind": "Job",
      "metadata": {
        "namespace": "{{ namespace }}",
        "generateName": "{{ name }}-",
        "labels": "{{ labels }}",
        "annotations": "{{ annotations }}"
      },
      "spec": {
        "backoffLimit": "{{ backoff_limit }}",
        "ttlSecondsAfterFinished": "{{ finished_job_ttl }}",
        "template": {
          "metadata": {
            "labels": "{{ labels }}",
            "annotations": "{{ annotations }}"
          },
          "spec": {
            "parallelism": 1,
            "completions": 1,
            "restartPolicy": "Never",
            "serviceAccountName": "{{ service_account_name }}",
            "containers": [
              {
                "name": "prefect-job",
                "image": "{{ image }}",
                "imagePullPolicy": "{{ image_pull_policy }}",
                "args": "{{ command }}",
                "env": "{{ env }}"
              }
            ]
          }
        }
      }
    }
  }
}

Then set labels/annotations at deploy time or on the pool, for example: - labels: { "enableIdentityHelper": "true", "appIdentity": "prefect" } - and any annotations your injector requires Apply the template to your pool - CLI:

prefect work-pool update "my-k8s-pool" --base-job-template ./base-job-template.json

- Or edit the pool in the UI and paste the template. Why this works - Most sidecar injectors watch AdmissionReview requests for Pod objects created from Job.spec.template. They match on spec.template.metadata.{labels,annotations}. Without that section, your labels stay on the Job and the Pod is unlabelled, so no mutation occurs. References - Run on Kubernetes (work pools): Run on Kubernetes (work pools) - Customize base job templates: Customize base job templates If you share the exact label/annotation keys your identity helper needs (and whether it requires namespace/selector setup), I can tailor the JSON to your injector’s expectations.

Great question — those labels need to live on the Pod template metadata to trigger the injector. Put them here in your base job template: - Job metadata (optional, for consistency/visibility) - Pod template metadata (required for injection) Hardcoded example (showing exactly where to place them):

{
  "job_configuration": {
    "job_manifest": {
      "apiVersion": "batch/v1",
      "kind": "Job",
      "metadata": {
        "namespace": "{{ namespace }}",
        "generateName": "{{ name }}-",
        "labels": {
          "enableIdentityHelper": "true",
          "appIdentity": "prefect"
        }
      },
      "spec": {
        "backoffLimit": "{{ backoff_limit }}",
        "ttlSecondsAfterFinished": "{{ finished_job_ttl }}",
        "template": {
          "metadata": {
            "labels": {
              "enableIdentityHelper": "true",
              "appIdentity": "prefect"
            }
          },
          "spec": {
            "parallelism": 1,
            "completions": 1,
            "restartPolicy": "Never",
            "serviceAccountName": "{{ service_account_name }}",
            "containers": [
              {
                "name": "prefect-job",
                "image": "{{ image }}",
                "imagePullPolicy": "{{ image_pull_policy }}",
                "args": "{{ command }}",
                "env": "{{ env }}"
              }
            ]
          }
        }
      }
    }
  }
}

If you prefer to keep the template generic and pass labels via variables, keep:

"metadata": { "labels": "{{ labels }}" },
"spec": { "template": { "metadata": { "labels": "{{ labels }}" }, ... } }

and set labels to include:

{
  "enableIdentityHelper": "true",
  "appIdentity": "prefect"
}

(e.g., as defaults in

variables.properties.labels.default

, or via work pool/deployment labels). Key point: spec.template.metadata.labels must include these keys — job-level labels alone won’t trigger the pod-level injector. If your injector uses annotations instead, mirror the same structure under

annotations

.

Short answer: only if your injector requires it. Prefect doesn’t care about an app label, but many mutating webhooks use a label selector (e.g., app or app.kubernetes.io/name). If your identity helper is matching on that, add it to the Pod template metadata (spec.template.metadata.labels) with a stable value. Use a constant value, not the Job name. Admission selectors should not depend on the generated Job/Pod names. Pick something consistent with your policy, e.g.: - app: prefect - or app: prefect-job - or use the recommended Kubernetes label: app.kubernetes.io/name: prefect Example (showing exactly where to put it alongside your existing labels):

"metadata": {
  "namespace": "{{ namespace }}",
  "generateName": "{{ name }}-",
  "labels": {
    "enableIdentityHelper": "true",
    "appIdentity": "prefect",
    "app": "prefect"                     // optional, only if your injector requires it
  }
},
"spec": {
  "template": {
    "metadata": {
      "labels": {
        "enableIdentityHelper": "true",
        "appIdentity": "prefect",
        "app": "prefect"                 // ← required here for pod-level injection
      }
    },
    "spec": {
      "containers": [
        {
          "name": "prefect-job",
          "image": "{{ image }}",
          "args": "{{ command }}",
          "env": "{{ env }}"
        }
      ]
    }
  }
}

If you’d rather keep the template generic, keep: - metadata.labels: "{{ labels }}" - spec.template.metadata.labels: "{{ labels }}" …and set labels to include app: prefect in your pool/deployment configuration. Tip: - Some injectors key off annotations instead of labels. If yours does, mirror the same placement under spec.template.metadata.annotations. - Prefect generates names for Jobs/Pods, but labels are independent—use a constant label for matching. Docs: - Customize base job templates: Customize base job templates - Run on Kubernetes (work pools): Run on Kubernetes (work pools)

Short answer: No. The worker pod and each flow-run Job pod are different pods; a sidecar only shares process space/volumes with containers in the same pod. A Job created by the Prefect Kubernetes work pool cannot “use” the Kerberos sidecar that’s running in the worker pod. What to do instead - Inject the Kerberos sidecar into the Job’s pod: - Add your injector’s labels/annotations to the Job’s pod template metadata so the mutating webhook fires:

"spec": {
      "template": {
        "metadata": {
          "labels": {
            "enableIdentityHelper": "true",
            "appIdentity": "prefect"
            // add "app": "prefect" if your injector matches on it
          },
          "annotations": {
            // add any injector-required annotations here
          }
        },
        "spec": { ... }
      }
    }

- Or explicitly add a Kerberos sidecar/initContainer and required volumes in the base job template for the pool. - Do not rely on the worker pod’s environment or sidecars to “carry over” to the Job pod—nothing from the worker pod is automatically shared with Job pods (env, volumes, processes, ticket caches). If you share which injector you’re using (and whether it matches on labels or annotations), I can provide a ready-to-paste base job template snippet. References - Customize base job templates: Customize base job templates - Run on Kubernetes (work pools): Run on Kubernetes (work pools)

You’re exactly right: in Prefect’s Kubernetes worker, label and annotation keys are “slugified,” which includes lowercasing. That’s why

enableIdentityHelper

becomes

enableidentityhelper

, and your injector doesn’t match. What’s happening - Prefect runs all label and annotation keys through a slugifier to keep them Kubernetes‑compliant. This uses python-slugify with the default lowercase behavior. - Code references: - Key slugification: prefect_kubernetes/utilities.py (

_slugify_label_key

) - Where it’s applied: prefect_kubernetes/worker.py (

_slugify_labels

,

_slugify_annotations

, and propagation to pod metadata) - Note: Label values preserve casing in recent 3.x releases, but keys are still lowercased. You’re on 3.6.22, which has this behavior. Why this breaks injectors - Many sidecar injectors match on an exact key (e.g.,

enableIdentityHelper

). Since Prefect lowercases the key to

enableidentityhelper

, the injector selector doesn’t fire. Can I bypass this by hardcoding the label in the base job template? - No. Even labels you place under

spec.template.metadata.labels

in the base job template are merged and passed back through the slugifier by the worker, so keys will be lowercased there too. Ways to resolve - Preferred: Use a lowercase key variant if your injector can accept it (e.g., configure it to look for

enableidentityhelper

instead of

enableIdentityHelper

), or use an alternative key it already supports that’s lowercase. - If your injector supports annotations and accepts lowercase keys there, move the trigger to an annotation. Be aware: Prefect also lowercases annotation keys, so the injector must accept the lowercase key. - If the injector requires the exact camelCase label/annotation key and cannot be changed: - Don’t rely on the injector for Jobs. Instead, explicitly add the Kerberos init/sidecar (and any volumes/volumeMounts) in the work pool’s base job template. This avoids key‑matching entirely and is the most reliable path with Prefect today. Docs: Customize base job templates. - If you want injector-based behavior long-term: - Consider updating the injector’s webhook to match case-insensitively or to recognize the lowercased key as an alias. - If this is not feasible, I recommend opening a Prefect feature request to allow opting out of key lowercasing (or a “raw passthrough” for keys). I’m happy to file one on your behalf if you share any constraints/requirements. If you can share which injector you’re using (and whether it can be configured to match lowercase keys or annotations), I can suggest the cleanest path or craft an explicit base job template snippet that bakes in the Kerberos sidecar/init logic.

@Marvin How to enable otel for prefect 3 workers? if i enable that will it give info (traces and metrics ) about the info the task and flows we are running in prefect?

Great question. Assuming you’re on Prefect 3.x: Short answer - Yes: you can enable OpenTelemetry for workers. You’ll get - Traces for every flow run and task run (names, IDs, state changes, exceptions, parent/child spans) - Resource metrics for the flow-run process (CPU and memory) - Traces do not auto-export by default; you must configure an OTel exporter in your worker environment. Metrics can export via standard OTLP env vars. How to enable in workers 1) Install OTel bits into the worker runtime

pip install "prefect[otel]"
# or at minimum:
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

2) Turn on Prefect’s built-in span creation and metrics - These are on by default, but you can set explicitly:

# Traces (spans for flows and tasks)
PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY=true

# Resource metrics (CPU / memory for the flow-run process)
PREFECT_TELEMETRY_ENABLE_RESOURCE_METRICS=true
PREFECT_TELEMETRY_RESOURCE_METRICS_INTERVAL_SECONDS=10

3) Point metrics to your OTLP endpoint - Prefect’s resource metrics exporter follows standard OTel env vars:

# Either a base endpoint…
OTEL_EXPORTER_OTLP_ENDPOINT=<http://otel-collector:4318>
# …or a metrics-specific one
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=<http://otel-collector:4318/v1/metrics>

4) Configure a trace exporter (required for traces to appear in your backend) By default, Prefect creates spans but does not wire up a trace exporter. Add one at process startup (e.g., in your flow’s entrypoint, a small prestart module, or a sitecustomize.py in your worker image):

# otel_init.py
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
import os

endpoint = os.getenv("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT") or \
           os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "<http://otel-collector:4318>")

resource = Resource.create({
    "service.name": os.getenv("OTEL_SERVICE_NAME", "prefect-worker"),
    # add useful context for filtering in your backend
    "deployment": os.getenv("PREFECT__DEPLOYMENT__NAME", ""),
    "work_pool": os.getenv("PREFECT__WORK_POOL__NAME", ""),
})

provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=f"{endpoint.rstrip('/')}/v1/traces"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

Import this as early as possible in your worker/flow process, before your flows run:

import otel_init  # noqa: F401  # ensure this runs before running flows/tasks

Where to set this up in workers - Docker/Kubernetes workers: add the env vars to your work pool’s base job template and bake otel_init.py into your image. Import it from your flow entrypoint or via sitecustomize. - Process/Local workers: set the env vars in the worker environment and ensure otel_init.py is imported before flows execute. What you’ll see - Traces: - Spans per flow run and task run - Attributes like flow/task names, run IDs, tags; events on state changes; exceptions on failure - Parent/child relationships across subflows/tasks - Metrics: - Process-level CPU and memory for the flow-run subprocess - Note: metrics are not per-task; they’re for the flow-run process Notes and limitations - Resource metrics are collected from the flow-run subprocess, so you’ll see them when running via workers/serve; not when running flows inline in the same process as your Python interpreter. - If you’re sending to a custom collector (Jaeger, Grafana Tempo, etc.), use the OTLP HTTP endpoint (e.g., http://collector:4318). Prefect won’t attach any Prefect API credentials to custom endpoints.

- If you prefer not to write code for tracing setup, you can start your worker process using OTel’s autoinstrumentation launcher so it installs a global tracer provider; otherwise, use the code snippet above. Docs and references - Telemetry concepts: Prefect Telemetry (concepts) - API: RunTelemetry and Telemetry settings If you can share where you want to send OTel data (e.g., Jaeger, Tempo, Datadog, OTEL Collector address) and which worker type you’re using (process, Docker, Kubernetes), I can give you a drop-in config tailored to your setup.

Thanks for the details — here’s how I’d approach this. What “exit status -6” usually means - A negative exit code generally maps to a Unix signal; -6 is SIGABRT. - SIGABRT typically indicates a hard abort from a native library (C/C++), e.g.: - assert() failures - heap corruption/double free detected by glibc - native extensions (NumPy/Arrow/Pandas/Torch/etc.) calling abort() on fatal errors - It’s different from OOM kills (you’d typically see exit code 137 or reason OOMKilled/SIGKILL 9) or segfaults (often -11/SIGSEGV). How to debug when there are few/no logs 1) Inspect the Pod/Container termination details - Find the pod for the run (you can use either the job-name or Prefect’s labels):

kubectl get pods -n <namespace> -l job-name=<prefect-generated-job-name> -o wide
# or, if you have the flow run ID:
kubectl get pods -n <namespace> -l "<http://prefect.io/flow-run-id=<FLOW_RUN_ID>|prefect.io/flow-run-id=<FLOW_RUN_ID>>"

- Describe and get full YAML to see termination reason, message, and signal:

kubectl describe pod <pod> -n <namespace>
kubectl get pod <pod> -n <namespace> -o yaml
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses[0].state.terminated.reason}{" "}{.status.containerStatuses[0].state.terminated.exitCode}{" "}{.status.containerStatuses[0].state.terminated.signal}{"\n"}{.status.containerStatuses[0].state.terminated.message}{"\n"}{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}'

- If your pod used restartPolicy=OnFailure and restarted, check the previous logs:

kubectl logs <pod> -n <namespace> --previous

- Check events for eviction, node pressure, or image/runtime errors:

kubectl get events -n <namespace> --sort-by=.lastTimestamp | grep <pod>

2) Check actual container logs (they may not have reached Prefect) - If the abort happens before the Prefect engine initializes, Prefect’s own logs can be empty. Always pull raw container logs via kubectl as above. 3) Turn on extra diagnostics in your job - Enable Prefect debug logs:

PREFECT_LOGGING_LEVEL=DEBUG

- Enable Python faulthandler to dump tracebacks on fatal signals (including SIGABRT) to stderr:

PYTHONFAULTHANDLER=1

- If you suspect native memory errors, enable allocator diagnostics:

PYTHONMALLOC=debug
MALLOC_CHECK_=2

These can surface useful messages right before the abort. 4) Watch for common native-library pitfalls - ABI/GLIBC mismatches between your image and the node - CPU instruction set mismatches (less likely for SIGABRT; SIGILL is more common there) - Mixed binary wheels (e.g., combining different manylinux variants) - Large native allocations (e.g., Arrow, Torch) that fail abruptly Do Kubernetes jobs restart automatically? - By default with a Prefect Kubernetes work pool: - restartPolicy: Never - backoffLimit: 0 - Result: no container restarts and no new pod retries by Kubernetes - You have three retry layers you can choose from: 1) Kubernetes Job retries (new pod on failure) - Increase backoffLimit in your work pool’s base job config/variables:

{
  "spec": {
    "backoffLimit": 3
  }
}

Kubernetes will create up to 3 new pods if the previous one fails. 2) Pod-internal restarts (same pod restarts the container) - Customize your base job template to use:

{
  "spec": {
    "backoffLimit": 0,
    "template": {
      "spec": {
        "restartPolicy": "OnFailure"
      }
    }
  }
}

This restarts the container within the same pod on failure. 3) Prefect-level retries (recommended for idempotent flows/tasks) - Add retries to your flow and/or tasks so Prefect can reschedule on crash:

from prefect import flow, task

     @task(retries=3, retry_delay_seconds=30)
     def my_task():
         ...

     @flow(retries=2, retry_delay_seconds=60)
     def my_flow():
         my_task()

Notes on eviction/rescheduling

- When backoffLimit is 0, Prefect’s Kubernetes integration configures the flow-run process to reschedule on SIGTERM (common on eviction). That won’t help for SIGABRT (-6), which is a hard crash; you’ll need one of the retry options above. If you can share: - The job/pod name and namespace - containerStatuses termination details (reason/exitCode/signal) - your work pool’s job template (especially restartPolicy/backoffLimit) - the container image and any heavy native libs you’re using …I can help pinpoint the most likely cause and provide a concrete job template and retry setup for your pool.

If you’re using HA self-hosted Prefect with Redis messaging, the dead-letter queue is in Redis, not in the Prefect API database. Short answer: yes — failed event messages can end up in a Redis-backed DLQ, and you inspect it with

redis-cli

. Where to look - Redis set key:

dlq

- Individual messages: keys like

dlq:<uuid>

Useful commands

bash
# how many failed messages are in the DLQ
redis-cli SCARD dlq

# list message keys
redis-cli SMEMBERS dlq

# inspect one message
redis-cli HGET dlq:<message_id> data

If you want to inspect all of them:

bash
redis-cli SMEMBERS dlq | while read msg_id; do
  echo "=== $msg_id ==="
  redis-cli HGET "$msg_id" data
done

If

jq

is available:

bash
redis-cli SMEMBERS dlq | while read msg_id; do
  echo "=== $msg_id ==="
  redis-cli HGET "$msg_id" data | jq .
done

A few important details - This DLQ is part of the Redis messaging layer used in HA setups. - Messages are sent there after retry attempts are exhausted. - The Prefect event persister itself does not have a separate DLQ in the API DB; if persistence keeps failing beyond its retry threshold, those events are typically only reflected in logs. What to verify in your setup For HA Redis-backed events, you should be using Redis messaging settings like: -

PREFECT_MESSAGING_BROKER="prefect_redis.messaging"

-

PREFECT_MESSAGING_CACHE="prefect_redis.messaging"

If you’re not using the Redis broker and are on the default in-memory broker, the DLQ is instead on disk under:

bash
~/.prefect/dlq

So in your case: - HA + Redis configured correctly → inspect Redis key

dlq

- non-Redis/default broker → inspect

~/.prefect/dlq

If you want, I can also help you with: - a kubectl command to inspect this in your Prefect server pod - a small script to replay/export DLQ messages - how to confirm from config whether your API is actually using

prefect_redis.messaging

Absolutely — here’s a practical guide for all three cases: 1. How to confirm which messaging backend you’re using 2. How to inspect the DLQ in Redis for HA/self-hosted 3. How to inspect the filesystem DLQ if you’re on the default in-memory broker 4. How to check Kubernetes pods 5. How to export / review / clean DLQ messages 6. What logs to check when events are being dropped --- 1) Confirm whether Prefect is actually using Redis messaging For HA, you generally want the API/server environment to include:

bash
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering

You can check this in a few ways. If running in Kubernetes

bash
kubectl get deploy -n <namespace>
kubectl describe deploy <prefect-server-deployment> -n <namespace>

Then look for env vars in the output: -

PREFECT_MESSAGING_BROKER

-

PREFECT_MESSAGING_CACHE

-

PREFECT_SERVER_EVENTS_CAUSAL_ORDERING

A more direct version:

bash
kubectl get deploy <prefect-server-deployment> -n <namespace> -o yaml | grep -A2 -E 'PREFECT_MESSAGING_BROKER|PREFECT_MESSAGING_CACHE|PREFECT_SERVER_EVENTS_CAUSAL_ORDERING'

If your deployment has multiple containers:

bash
kubectl get pod -n <namespace>
kubectl describe pod <pod-name> -n <namespace>

If running with Docker

bash
docker inspect <container-name> | grep -A2 -E 'PREFECT_MESSAGING_BROKER|PREFECT_MESSAGING_CACHE|PREFECT_SERVER_EVENTS_CAUSAL_ORDERING'

From inside a running container/pod

bash
env | grep PREFECT_MESSAGING
env | grep PREFECT_SERVER_EVENTS_CAUSAL_ORDERING

Interpretation - If you see

prefect_redis.messaging

→ you’re using Redis-backed messaging, so inspect Redis DLQ. - If not, you may still be on the default memory broker, so inspect

~/.prefect/dlq

. --- 2) Redis DLQ inspection for HA setup If you’re using Redis messaging, the dead-letter queue is in Redis. Check whether there are failed messages

bash
redis-cli SCARD dlq

If that returns

0

, there are currently no dead-lettered messages. List all failed message keys

bash
redis-cli SMEMBERS dlq

You’ll see items like:

bash
dlq:abc123...
dlq:def456...

Inspect one message

bash
redis-cli HGET dlq:<message_id> data

Example:

bash
redis-cli HGET dlq:abc123 data

Pretty-print with jq

bash
redis-cli HGET dlq:abc123 data | jq .

Inspect all messages

bash
redis-cli SMEMBERS dlq | while read msg_id; do
  echo "=== $msg_id ==="
  redis-cli HGET "$msg_id" data
done

With `jq`:

bash
redis-cli SMEMBERS dlq | while read msg_id; do
  echo "=== $msg_id ==="
  redis-cli HGET "$msg_id" data | jq .
done

Count and summarize quickly

bash
echo "DLQ count: $(redis-cli SCARD dlq)"

--- 3) Redis DLQ inspection from Kubernetes If Redis is running as a pod/service in your cluster, you can exec into the Redis pod. Find Redis pod

bash
kubectl get pods -n <namespace>

Exec into Redis pod

bash
kubectl exec -it <redis-pod> -n <namespace> -- redis-cli SCARD dlq

List DLQ keys

bash
kubectl exec -it <redis-pod> -n <namespace> -- redis-cli SMEMBERS dlq

Inspect a message

bash
kubectl exec -it <redis-pod> -n <namespace> -- redis-cli HGET dlq:<message_id> data

If your Redis requires host/port instead of local socket:

bash
kubectl exec -it <prefect-api-pod> -n <namespace> -- sh
redis-cli -h <redis-service-name> -p 6379 SCARD dlq
redis-cli -h <redis-service-name> -p 6379 SMEMBERS dlq

If Redis requires auth:

bash
redis-cli -h <redis-host> -p 6379 -a '<password>' SCARD dlq

--- 4) Export Redis DLQ messages to a file This is useful if you want to inspect them safely before deleting anything.

bash
redis-cli SMEMBERS dlq | while read msg_id; do
  redis-cli HGET "$msg_id" data
done > prefect_dlq_dump.jsonl

If you want message ids alongside payloads: ```bash

redis-cli SMEMBERS dlq | while read msg_id; do echo "{\"redis_key\":\"$msg_id\",\"payload\":$(redis-cli HGET "$msg_id" data)}" done > prefect_dlq_dump.jsonl

If the payload is already JSON, this can work nicely for analysis.

---

*5) Remove / clean Redis DLQ messages*

Only do this after you’ve exported or reviewed them.

*Delete one message*
```bash
redis-cli DEL dlq:<message_id>
redis-cli SREM dlq dlq:<message_id>

Delete all DLQ message entries

bash
redis-cli SMEMBERS dlq | xargs redis-cli DEL
redis-cli DEL dlq

If

xargs

is not available or message count is small:

bash
for key in $(redis-cli SMEMBERS dlq); do
  redis-cli DEL "$key"
done
redis-cli DEL dlq

--- 6) Filesystem DLQ inspection for non-Redis/default broker If your server is not using Redis messaging, failed messages go to a local DLQ directory under Prefect home. Usually:

bash
~/.prefect/dlq

Sometimes in containers this may resolve under the container user’s home, so check:

bash
echo $HOME
ls -lah ~/.prefect
ls -lah ~/.prefect/dlq

List messages

bash
ls -lah ~/.prefect/dlq

Inspect one file

bash
cat ~/.prefect/dlq/<message_file>

Pretty-print one file

bash
jq . ~/.prefect/dlq/<message_file>

Inspect all files

bash
for f in ~/.prefect/dlq/*; do
  echo "=== $f ==="
  cat "$f"
done

With formatting:

bash
for f in ~/.prefect/dlq/*; do
  echo "=== $f ==="
  jq . "$f"
done

Count messages

bash
ls -1 ~/.prefect/dlq | wc -l

Archive them before cleanup

bash
mkdir -p ~/.prefect/dlq_backup
cp ~/.prefect/dlq/* ~/.prefect/dlq_backup/

Delete one file

bash
rm ~/.prefect/dlq/<message_file>

Delete all files

bash
rm -f ~/.prefect/dlq/*

--- 7) Filesystem DLQ inspection in Kubernetes If you suspect the API pod is using the memory broker, exec into the Prefect pod:

bash
kubectl exec -it <prefect-api-pod> -n <namespace> -- sh

Then:

bash
echo $HOME
ls -lah ~/.prefect
ls -lah ~/.prefect/dlq

If the container runs under a different home path, search:

bash
find / -type d -name dlq 2>/dev/null

Or:

bash
find / -path '*/.prefect/dlq' 2>/dev/null

--- 8) Check Prefect logs for dropped or dead-lettered events There are two important failure modes: Message broker / consumer failures These may lead to DLQ entries. Search logs for:

bash
kubectl logs <prefect-api-pod> -n <namespace> | grep -i "dead letter"
kubectl logs <prefect-api-pod> -n <namespace> | grep -i "sending to dead letter queue"
kubectl logs <prefect-api-pod> -n <namespace> | grep -i "could not be processed after"

Event persister failures These do not necessarily create DLQ entries. Events can be dropped after retries. Search for:

bash
kubectl logs <prefect-api-pod> -n <namespace> | grep -i "dropping"
kubectl logs <prefect-api-pod> -n <namespace> | grep -i "max flush retries"
kubectl logs <prefect-api-pod> -n <namespace> | grep -i "event queue"

If using Docker:

bash
docker logs <prefect-container> 2>&1 | grep -i "dead letter"
docker logs <prefect-container> 2>&1 | grep -i "max flush retries"
docker logs <prefect-container> 2>&1 | grep -i "dropping"

--- 9) Quick health-check flow I’d use in HA If I were troubleshooting this in your HA self-hosted setup, I’d do:

bash
# 1. confirm Prefect is configured to use Redis messaging
kubectl get deploy <prefect-server-deployment> -n <namespace> -o yaml | grep -A2 -E 'PREFECT_MESSAGING_BROKER|PREFECT_MESSAGING_CACHE|PREFECT_SERVER_EVENTS_CAUSAL_ORDERING'

# 2. check whether DLQ has messages
kubectl exec -it <redis-pod> -n <namespace> -- redis-cli SCARD dlq

# 3. if > 0, inspect them
kubectl exec -it <redis-pod> -n <namespace> -- redis-cli SMEMBERS dlq

# 4. inspect Prefect API logs for why processing failed
kubectl logs <prefect-api-pod> -n <namespace> | grep -iE 'dead letter|dropping|max flush retries|redis'

--- 10) Small script: inspect Redis DLQ nicely ```bash #!/bin/sh

echo "DLQ count: $(redis-cli SCARD dlq)" echo for msg_id in $(redis-cli SMEMBERS dlq); do echo "=== $msg_id ===" redis-cli HGET "$msg_id" data | jq . 2>/dev/null || redis-cli HGET "$msg_id" data echo done

---

*11) Small script: inspect filesystem DLQ nicely*

```bash
#!/bin/sh

DLQ_PATH="${HOME}/.prefect/dlq"

if [ ! -d "$DLQ_PATH" ]; then
  echo "DLQ path not found: $DLQ_PATH"
  exit 1
fi

echo "DLQ files count: $(ls -1 "$DLQ_PATH" 2>/dev/null | wc -l)"
echo

for f in "$DLQ_PATH"/*; do
  [ -e "$f" ] || continue
  echo "=== $f ==="
  jq . "$f" 2>/dev/null || cat "$f"
  echo
done

--- 12) Important note on replaying messages I don’t see a built-in supported Prefect command for “replay DLQ messages” directly. In practice, operators usually: - inspect/export the failed messages - identify the root cause - clear the DLQ - let fresh events flow normally If you need replay behavior, that would usually require a custom script and careful validation of payload format. --- Relevant references - Prefect docs: Prefect documentation - Prefect repo: PrefectHQ/prefect - Prefect Redis integration package lives in the Prefect repo and provides

prefect_redis.messaging

If you want, I can next give you a copy-paste exact checklist for Kubernetes HA with placeholders like: -

<namespace>

-

<prefect-api-deployment>

-

<redis-pod>

so you can run it step by step in your cluster.

If

SCARD dlq

and

SMEMBERS dlq

return nothing, the most likely possibilities are: 1. There are currently no DLQ messages 2. You are connected to the wrong Redis instance / DB 3. Prefect is not actually using Redis messaging 4. The failure is happening later in the pipeline, so events are being dropped in logs rather than sent to the Redis DLQ Let’s narrow it down. First: what exactly do you see? Please share the raw output of these:

bash
redis-cli SCARD dlq
redis-cli SMEMBERS dlq
redis-cli INFO keyspace

Because: -

SCARD dlq

returning

0

means the key exists but is empty -

(nil)

or blank can mean the key does not exist -

INFO keyspace

helps confirm which Redis DB has keys --- Check that you are on the correct Redis DB Sometimes Prefect is using Redis DB

1

or another DB, while

redis-cli

defaults to DB

0

. Try:

bash
redis-cli -n 0 SCARD dlq
redis-cli -n 1 SCARD dlq
redis-cli -n 2 SCARD dlq
redis-cli -n 3 SCARD dlq

Also:

bash
redis-cli -n 0 DBSIZE
redis-cli -n 1 DBSIZE
redis-cli -n 2 DBSIZE
redis-cli -n 3 DBSIZE

If one DB has keys and the others don’t, inspect that DB further. You can also search for likely Prefect keys:

bash
redis-cli --scan | grep -E 'dlq|event|prefect'

Or for a specific DB:

bash
redis-cli -n 0 --scan | grep -E 'dlq|event|prefect'

--- Check whether Prefect is really configured to use Redis messaging If Prefect is not using

prefect_redis.messaging

, then the Redis DLQ will stay empty forever. From the Prefect API/server pod/container:

bash
env | grep PREFECT_MESSAGING
env | grep PREFECT_SERVER_EVENTS_CAUSAL_ORDERING

You want to see:

bash
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering

If you do not see those, then you’re probably not using the Redis event broker. In Kubernetes:

bash
kubectl exec -it <prefect-api-pod> -n <namespace> -- env | grep PREFECT_MESSAGING
kubectl exec -it <prefect-api-pod> -n <namespace> -- env | grep PREFECT_SERVER_EVENTS_CAUSAL_ORDERING

--- If not using Redis messaging, check the filesystem DLQ instead Exec into the API pod and inspect:

bash
kubectl exec -it <prefect-api-pod> -n <namespace> -- sh
ls -lah ~/.prefect/dlq

If needed:

bash
find / -path '*/.prefect/dlq' 2>/dev/null

--- Check if the issue is event dropping, not DLQ A very important detail: some failures do not end up in the Redis DLQ. If the event persister fails repeatedly, Prefect may drop events and only log that. Search API/server logs:

bash
kubectl logs <prefect-api-pod> -n <namespace> | grep -iE 'dead letter|dropping|max flush retries|redis|event'

Especially look for: -

sending to dead letter queue

-

Message could not be processed after

-

Max flush retries

-

dropping

If you see dropping-related logs, then your missing events may never have reached the Redis DLQ. --- Check whether Prefect is writing anything at all into Redis If Prefect is using Redis messaging correctly, you should often see stream/group-related keys too, not just

dlq

. Try:

bash
redis-cli --scan | grep -E 'events|stream|consumer|dlq'

And:

bash
redis-cli TYPE dlq

If Redis has no Prefect-related keys at all, either: - wrong Redis host - wrong DB - Prefect not configured for Redis messaging - Prefect cannot connect to Redis --- Best next commands Please run these and paste the output:

bash
redis-cli SCARD dlq
redis-cli SMEMBERS dlq
redis-cli TYPE dlq
redis-cli INFO keyspace
redis-cli --scan | grep -E 'dlq|event|prefect'

And from the Prefect API pod:

bash
env | grep PREFECT_MESSAGING
env | grep PREFECT_SERVER_EVENTS_CAUSAL_ORDERING

If you want, paste those outputs here and I’ll tell you exactly which case you’re in.

I couldn’t find an existing GitHub issue or discussion matching that exact React UI error message. What I did find is that there are several known UI/custom-run parameter issues in the new UI, especially around parameter schema handling and

$ref

-based schemas: * Prefect blocks are not supported as flow run parameters * [UI v2] Custom run parameter values not preserved * Prefect Cloud UI parameters missing type hints Your error:

cannot use 'IN' operator to search for $ref in undefined

looks like a frontend schema/parsing bug in the custom run form, likely triggered by a deployment parameter schema the UI doesn’t fully handle. A few quick things to check: * Does it happen only for one deployment, or all deployments? * Are you using complex parameter types like Pydantic models, nested objects, unions, dataclasses, blocks, or optional structured params? * Does switching to the raw JSON parameters editor avoid the issue? If you want, I can help narrow it down and draft a minimal repro. If you’d like to raise a ticket, the best path is: * Prefect GitHub Issues If you share: * your Prefect version * whether this is Cloud or self-hosted Server * the flow parameter types / schema …I can help determine whether this matches one of the known schema-related bugs or help you write a stronger bug report.