Great question. Here’s a quick, reliable way to set up logging when running Prefect on Kubernetes (assuming Prefect 3.x). If any of this doesn’t match what you want (e.g., JSON logs, Datadog/CloudWatch export), let me know and I’ll tailor it. The two most common setups A) Stream logs to Prefect Cloud (recommended) - Make sure your Kubernetes worker is connected to Cloud: - Set

PREFECT_API_URL

and

PREFECT_API_KEY

on the worker Pod. - Prefect will automatically stream flow/task logs to Cloud; no extra config needed. - Adjust verbosity with environment variables (on your work pool, job, or worker):

PREFECT_LOGGING_LEVEL=INFO          # or DEBUG
  PREFECT_LOGGING_LOG_PRINTS=true     # capture print() as logs

- View logs: - In the Cloud UI on the flow run page - Or via CLI:

prefect flow-run logs <FLOW_RUN_ID>

(Run

prefect flow-run logs --help

for options.) B) Ship logs to your cluster’s logging stack (e.g., Datadog, CloudWatch, GCP) - Prefect emits logs to the container’s stdout/stderr, so any node-level log collector (Fluent Bit/Fluentd/Vector/etc.) or sidecar will pick them up. - Ensure your Kubernetes worker keeps “streaming pod output” enabled (it’s on by default) so you also see logs in the worker/Cloud while the run executes. - Optional: add Pod annotations your log collector expects (examples below). Where to set environment variables - In Prefect Cloud → Work Pools → your Kubernetes pool: - “Environment Variables” (applies to all jobs) - Or in your deployment’s

job_variables.env

(applies to that deployment) - Example in `prefect.yaml`:

deployments:
    - name: k8s-demo
      work_pool:
        name: kubernetes
        job_variables:
          env:
            PREFECT_LOGGING_LEVEL: "INFO"
            PREFECT_LOGGING_LOG_PRINTS: "true"
            # Only needed if not already on the worker:
            # PREFECT_API_URL: "<https://api.prefect.cloud/api>"
            # PREFECT_API_KEY: "<your key>"

JSON/structured logs (optional) - If your logging backend prefers JSON, you can switch Prefect’s console logs to JSON with a custom logging config. - High-level steps: - Build an image (or mount a ConfigMap) with a

logging.yml

that uses Prefect’s JsonFormatter - Point Prefect to it via

PREFECT_LOGGING_SETTINGS_PATH=/path/to/logging.yml

- If you want, I can share a minimal

logging.yml

to enable JSON to stdout. Kubernetes annotations (optional) - Some log collectors key off annotations on Pods. Add these in your Kubernetes work pool base job template or via `job_variables`:

job_variables:
    pod:
      metadata:
        annotations:
          # Datadog example
          <http://ad.datadoghq.com/prefect-job.logs|ad.datadoghq.com/prefect-job.logs>: '[{"service":"prefect-flow","source":"kubernetes"}]'
          # Or GCP / Splunk vendor-specific annotations if required

Quick checklist - Use

get_run_logger()

in your flow/tasks to emit logs. - Set

PREFECT_LOGGING_LEVEL

and

PREFECT_LOGGING_LOG_PRINTS

as needed. - Make sure your worker has `PREFECT_API_URL`/`PREFECT_API_KEY` if using Cloud. - Let your cluster’s log agent collect container stdout/stderr if you want external aggregation. - Use the Prefect UI or

prefect flow-run logs <id>

to view logs during/after runs. Questions for you - Do you want logs only in Prefect Cloud, or also in Datadog/CloudWatch/ELK, etc.? - Do you need JSON formatting? - Are you configuring via Prefect Cloud Work Pools or via code/CI (i.e.,

prefect.yaml

)?