Sebastian S
11/17/2025, 2:01 PMMarvin
11/17/2025, 2:03 PMMarvin
11/17/2025, 2:06 PM~/.prefect/logging.yml (or anywhere and point an env var to it):
version: 1
disable_existing_loggers: false
formatters:
json:
'()': prefect.logging.formatters.JsonFormatter
fmt: default # or 'pretty' for multiline, but 'default' is best for ingestion
handlers:
console:
class: prefect.logging.handlers.PrefectConsoleHandler
stream: <ext://sys.stdout>
formatter: json
api:
class: prefect.logging.handlers.APILogHandler
loggers:
# Prefect's main loggers; attach both console (JSON to stdout) and API (Prefect UI)
prefect:
level: INFO
handlers: [console, api]
propagate: false
prefect.flow_runs:
level: INFO
handlers: [console, api]
propagate: false
prefect.task_runs:
level: INFO
handlers: [console, api]
propagate: false
root:
level: WARNING
handlers: [console]
2) Point Prefect to your logging config (if not using the default path)
- Linux/macOS:
export PREFECT_LOGGING_SETTINGS_PATH=/path/to/logging.yml
- Windows (PowerShell):
$env:PREFECT_LOGGING_SETTINGS_PATH="C:\path\to\logging.yml"
3) Run your flows — logs will now be emitted as JSON to stdout and still go to the Prefect UI if you kept the API handler.
Datadog notes
- The Datadog Agent automatically parses one-JSON-object-per-line logs. Just ensure it’s tailing your worker/process/container stdout with logs enabled.
- Add tags at the Agent level (recommended) or enrich log records with extra fields if you need custom keys.
- If you want only Datadog (and not Prefect UI) logs, remove the api handler from the logger(s).
References
- Prefect logging customization (3.x): Docs: Logging customization
- JSON formatter API: API: JsonFormatter
- Handlers API: API: logging handlers
- Datadog log collection + JSON parsing: Datadog: supported log formats
If you can share where your flows/workers are running (local, Docker, Kubernetes) and whether you want to keep logs in the Prefect UI, I can tailor the config (Agent setup, tags, etc.) for your environment.Sebastian S
11/17/2025, 2:22 PMSebastian S
11/17/2025, 2:31 PMMarvin
11/17/2025, 2:31 PMMarvin
11/17/2025, 2:42 PMkubectl -n <your-namespace> create configmap prefect-logging \
--from-file=logging.yml=./logging.yml
2) Configure flow-run pods via your work pool base job template
- Export the default template:
prefect work-pool get-default-base-job-template --type kubernetes -f base-template.json
- Edit base-template.json to add:
- a volume for the ConfigMap,
- a volumeMount for the container,
- an env var PREFECT_LOGGING_SETTINGS_PATH pointing to the mounted file.
Minimal example (only showing the parts to add/change):
{
"job_configuration": {
"job_manifest": {
"spec": {
"template": {
"spec": {
"volumes": [
{
"name": "prefect-logging",
"configMap": {
"name": "prefect-logging"
}
}
],
"containers": [
{
"name": "prefect-job",
"env": [
{
"name": "PREFECT_LOGGING_SETTINGS_PATH",
"value": "/etc/prefect/logging/logging.yml"
}
],
"volumeMounts": [
{
"name": "prefect-logging",
"mountPath": "/etc/prefect/logging",
"readOnly": true
}
]
}
]
}
}
}
}
}
}
- Apply it to your pool:
prefect work-pool update "<your-pool-name>" --base-job-template base-template.json
3) Mount the same ConfigMap in the worker Helm release
- In the Prefect Worker Helm chart, add:
- a volume from configMap: { name: prefect-logging },
- a volumeMount to the worker container (e.g., mountPath: /etc/prefect/logging),
- set PREFECT_LOGGING_SETTINGS_PATH=/etc/prefect/logging/logging.yml.
- Exact keys vary by chart version; if you share your values.yaml (or chart version), I’ll give you a drop-in snippet.
Option B (simpler ongoing): bake logging.yml into your image
- Copy logging.yml into your flow image at ~/.prefect/logging.yml (or any path).
- If you use ~/.prefect/logging.yml, Prefect will pick it up automatically (no env var needed).
- Otherwise set PREFECT_LOGGING_SETTINGS_PATH to that path in your work pool base job template (env) and optionally in the worker Helm values.
Datadog reminder
- With the JSON formatter, logs go to stdout as single JSON objects per line. Ensure your Datadog Agent is tailing the namespace/pods you care about; no special parsing config is needed beyond JSON auto-detection.
Helpful docs
- Logging customization: Prefect 3 logging customization
- Kubernetes work pool/base job template: Kubernetes work pools
- Helm charts: Prefect Helm repo
If you paste your values.yaml (or confirm which Prefect Helm chart/version you’re using), I’ll return an exact values patch for the worker.Sebastian S
11/17/2025, 3:54 PM---
apiVersion: <http://source.toolkit.fluxcd.io/v1|source.toolkit.fluxcd.io/v1>
kind: HelmRepository
metadata:
name: prefect-helm
namespace: flux-system
spec:
interval: 5m
url: <https://prefecthq.github.io/prefect-helm/>
timeout: 2m
---
apiVersion: v1
kind: Namespace
metadata:
name: prefect-dev
annotations:
<http://chicagotrading.io/otel-logging-datadog|chicagotrading.io/otel-logging-datadog>: "true"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: prefect-dev
namespace: prefect-dev
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
name: prefect-job-creator-dev
rules:
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRoleBinding
metadata:
name: prefect-job-creator-dev
subjects:
- kind: ServiceAccount
name: prefect-worker-eus2-reseng-dev
namespace: prefect-dev
- kind: ServiceAccount
name: prefect-worker-ncus-reseng-dev
namespace: prefect-dev
roleRef:
kind: ClusterRole
name: prefect-job-creator-dev
apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
---
apiVersion: <http://helm.toolkit.fluxcd.io/v2beta1|helm.toolkit.fluxcd.io/v2beta1>
kind: HelmRelease
metadata:
name: prefect-server-reseng-dev
namespace: prefect-dev
spec:
interval: 5m
install:
createNamespace: true
chart:
spec:
chart: prefect-server
version: "2025.11.5150433"
sourceRef:
kind: HelmRepository
name: prefect-helm
namespace: flux-system
interval: 1m
values:
global:
#prefect:
#image:
#repository: <http://artifactory.chicagotrading.com/docker-all/prefecthq/prefect-dev|artifactory.chicagotrading.com/docker-all/prefecthq/prefect-dev>
#prefectTag: sha-7cfd1c3-python3.11
#pullPolicy: IfNotPresent
env:
- name: PREFECT_LOGGING_EXTRA_LOGGERS
value: "[\"prefect.engine\",\"prefect.events\"]"
server:
debug: false
loggingLevel: INFO
replicaCount: 2
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi
uiConfig:
prefectUiApiUrl: <https://prefect-reseng-dev.chicagotrading.io/api>
# Prefect settings
# <https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-server/values.yaml#L36>
env:
- name: PREFECT_SERVER_UI_SHOW_PROMOTIONAL_CONTENT
value: "false"
# <https://docs.prefect.io/v3/concepts/caching> - see "Caching requires default persistence"
- name: PREFECT_RESULTS_PERSIST_BY_DEFAULT
value: "true"
- name: PREFECT_LOCAL_STORAGE_PATH
value: "/home/prefect/.prefect/storage"
# Postgres configuration
- name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
value: "20" # Originally 5
- name: PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW
value: "20" # Originally 10
- name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT
value: "60" # Originally 30
- name: PREFECT_SERVER_DATABASE_TIMEOUT
value: "30" # Originally 10
backgroundServices:
debug: false
loggingLevel: INFO
runAsSeparateDeployment: true
messaging:
broker: prefect_redis.messaging
cache: prefect_redis.messaging
redis:
# This should be managed by reseng team, need some extra terraform to allow it
# <https://chicagotrading.slack.com/archives/C07MJJ0CVNV/p1762363085022799?thread_ts=1762265780.557339&cid=C07MJJ0CVNV>
host: prefect-reseng-dev-redis-svc.prefect-dev.svc.cluster.local
port: 6379
db: 0
ssl: false
# Disable bundled PostgreSQL as we'll use Azure PostgreSQL
postgresql:
enabled: false
secret:
create: false
name: "prefect-postgresql-credentials"
ingress:
enabled: true
className: nginx
host:
hostname: <http://prefect-reseng-dev.chicagotrading.io|prefect-reseng-dev.chicagotrading.io>
tls: true
annotations:
<http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: chicagotradingio
extraPaths:
- path: /api
pathType: Prefix
backend:
service:
name: prefect-server
port:
number: 4200
# Set image registry to use Artifactory
image:
registry: <http://artifactory.chicagotrading.com/docker-all|artifactory.chicagotrading.com/docker-all>
---
apiVersion: <http://secrets.hashicorp.com/v1beta1|secrets.hashicorp.com/v1beta1>
kind: VaultStaticSecret
metadata:
name: prefect-postgresql-credentials
namespace: prefect-dev
spec:
destination:
create: true
name: prefect-postgresql-credentials
mount: azure_kv
path: research_engineering/azure/ncus/psql-admin-dev
refreshAfter: 30m
type: kv-v2
vaultAuthRef: prefect-dev
---
apiVersion: <http://secrets.hashicorp.com/v1beta1|secrets.hashicorp.com/v1beta1>
kind: VaultAuth
metadata:
name: prefect-dev
namespace: prefect-dev
spec:
kubernetes:
role: prefect-dev
serviceAccount: prefect-dev
tokenExpirationSeconds: 600
method: kubernetes
mount: kubernetesSebastian S
11/17/2025, 3:55 PMMarvin
11/17/2025, 3:55 PMMarvin
11/17/2025, 3:57 PMvalues.yaml (and which chart/version you’re using, e.g., prefect/prefect-worker)? I’ll tailor an exact patch.
In the meantime, here’s a working pattern you can adapt.
Step 1: Create a ConfigMap for logging.yml
kubectl -n <ns> create configmap prefect-logging \
--from-file=logging.yml=./logging.yml
Step 2: Make flow-run pods use it (via work pool base job template)
prefect work-pool get-default-base-job-template --type kubernetes -f base-template.json
Edit to add volume, mount, and env (showing only the added bits):
{
"job_configuration": {
"job_manifest": {
"spec": {
"template": {
"spec": {
"volumes": [
{
"name": "prefect-logging",
"configMap": {
"name": "prefect-logging"
}
}
],
"containers": [
{
"name": "prefect-job",
"env": [
{
"name": "PREFECT_LOGGING_SETTINGS_PATH",
"value": "/etc/prefect/logging/logging.yml"
}
],
"volumeMounts": [
{
"name": "prefect-logging",
"mountPath": "/etc/prefect/logging",
"readOnly": true
}
]
}
]
}
}
}
}
}
}
Apply it:
prefect work-pool update "<your-pool-name>" --base-job-template base-template.json
Step 3: Make the worker pod use it (Helm values)
The exact keys depend on your chart version. If your chart exposes extras like extraEnv, extraVolumes, and extraVolumeMounts, something like this works:
worker:
env:
- name: PREFECT_LOGGING_SETTINGS_PATH
value: /etc/prefect/logging/logging.yml
extraVolumes:
- name: prefect-logging
configMap:
name: prefect-logging
extraVolumeMounts:
- name: prefect-logging
mountPath: /etc/prefect/logging
readOnly: true
If your chart uses different keys (e.g., podTemplate, pod, container.env, etc.), send me your current values and I’ll map it precisely.
Notes
- You can skip setting the env var if you mount the file to ~/.prefect/logging.yml (Prefect picks it up automatically).
- The flow-run side must be configured via the work pool base job template; the Helm chart only affects the worker pod itself.
Refs
- Prefect Helm charts: prefect-helm
- Kubernetes work pools: Kubernetes work pools
- Logging customization: Logging customizationSebastian S
11/17/2025, 4:02 PM---
apiVersion: <http://source.toolkit.fluxcd.io/v1|source.toolkit.fluxcd.io/v1>
kind: HelmRepository
metadata:
name: prefect-helm
namespace: flux-system
spec:
interval: 5m
url: <https://prefecthq.github.io/prefect-helm/>
timeout: 2m
---
apiVersion: v1
kind: Namespace
metadata:
name: prefect-dev
annotations:
<http://chicagotrading.io/otel-logging-datadog|chicagotrading.io/otel-logging-datadog>: "true"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: prefect-dev
namespace: prefect-dev
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
name: prefect-job-creator-dev
rules:
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRoleBinding
metadata:
name: prefect-job-creator-dev
subjects:
- kind: ServiceAccount
name: prefect-worker-eus2-reseng-dev
namespace: prefect-dev
- kind: ServiceAccount
name: prefect-worker-ncus-reseng-dev
namespace: prefect-dev
roleRef:
kind: ClusterRole
name: prefect-job-creator-dev
apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
---
apiVersion: <http://helm.toolkit.fluxcd.io/v2beta1|helm.toolkit.fluxcd.io/v2beta1>
kind: HelmRelease
metadata:
name: prefect-server-reseng-dev
namespace: prefect-dev
spec:
interval: 5m
install:
createNamespace: true
chart:
spec:
chart: prefect-server
version: "2025.11.5150433"
sourceRef:
kind: HelmRepository
name: prefect-helm
namespace: flux-system
interval: 1m
values:
global:
#prefect:
#image:
#repository: <http://artifactory.chicagotrading.com/docker-all/prefecthq/prefect-dev|artifactory.chicagotrading.com/docker-all/prefecthq/prefect-dev>
#prefectTag: sha-7cfd1c3-python3.11
#pullPolicy: IfNotPresent
env:
- name: PREFECT_LOGGING_EXTRA_LOGGERS
value: "[\"prefect.engine\",\"prefect.events\"]"
server:
debug: false
loggingLevel: INFO
replicaCount: 2
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi
uiConfig:
prefectUiApiUrl: <https://prefect-reseng-dev.chicagotrading.io/api>
# Prefect settings
# <https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-server/values.yaml#L36>
env:
- name: PREFECT_SERVER_UI_SHOW_PROMOTIONAL_CONTENT
value: "false"
# <https://docs.prefect.io/v3/concepts/caching> - see "Caching requires default persistence"
- name: PREFECT_RESULTS_PERSIST_BY_DEFAULT
value: "true"
- name: PREFECT_LOCAL_STORAGE_PATH
value: "/home/prefect/.prefect/storage"
# Postgres configuration
- name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
value: "20" # Originally 5
- name: PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW
value: "20" # Originally 10
- name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT
value: "60" # Originally 30
- name: PREFECT_SERVER_DATABASE_TIMEOUT
value: "30" # Originally 10
backgroundServices:
debug: false
loggingLevel: INFO
runAsSeparateDeployment: true
messaging:
broker: prefect_redis.messaging
cache: prefect_redis.messaging
redis:
# This should be managed by reseng team, need some extra terraform to allow it
# <https://chicagotrading.slack.com/archives/C07MJJ0CVNV/p1762363085022799?thread_ts=1762265780.557339&cid=C07MJJ0CVNV>
host: prefect-reseng-dev-redis-svc.prefect-dev.svc.cluster.local
port: 6379
db: 0
ssl: false
# Disable bundled PostgreSQL as we'll use Azure PostgreSQL
postgresql:
enabled: false
secret:
create: false
name: "prefect-postgresql-credentials"
ingress:
enabled: true
className: nginx
host:
hostname: <http://prefect-reseng-dev.chicagotrading.io|prefect-reseng-dev.chicagotrading.io>
tls: true
annotations:
<http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: chicagotradingio
extraPaths:
- path: /api
pathType: Prefix
backend:
service:
name: prefect-server
port:
number: 4200
# Set image registry to use Artifactory
image:
registry: <http://artifactory.chicagotrading.com/docker-all|artifactory.chicagotrading.com/docker-all>
---
apiVersion: <http://secrets.hashicorp.com/v1beta1|secrets.hashicorp.com/v1beta1>
kind: VaultStaticSecret
metadata:
name: prefect-postgresql-credentials
namespace: prefect-dev
spec:
destination:
create: true
name: prefect-postgresql-credentials
mount: azure_kv
path: research_engineering/azure/ncus/psql-admin-dev
refreshAfter: 30m
type: kv-v2
vaultAuthRef: prefect-dev
---
apiVersion: <http://secrets.hashicorp.com/v1beta1|secrets.hashicorp.com/v1beta1>
kind: VaultAuth
metadata:
name: prefect-dev
namespace: prefect-dev
spec:
kubernetes:
role: prefect-dev
serviceAccount: prefect-dev
tokenExpirationSeconds: 600
method: kubernetes
mount: kubernetesMarvin
11/17/2025, 4:02 PMMarvin
11/17/2025, 4:04 PMapiVersion: v1
kind: ConfigMap
metadata:
name: prefect-logging
namespace: prefect-dev
data:
logging.yml: |
version: 1
disable_existing_loggers: false
formatters:
json:
'()': prefect.logging.formatters.JsonFormatter
fmt: default
handlers:
console:
class: prefect.logging.handlers.PrefectConsoleHandler
stream: <ext://sys.stdout>
formatter: json
api:
class: prefect.logging.handlers.APILogHandler
loggers:
prefect:
level: INFO
handlers: [console, api]
propagate: false
prefect.flow_runs:
level: INFO
handlers: [console, api]
propagate: false
prefect.task_runs:
level: INFO
handlers: [console, api]
propagate: false
root:
level: WARNING
handlers: [console]
2) Patch your HelmRelease values to mount and use it
You already set env for server and backgroundServices. Add these:
- For the Prefect Server API/UI/background pods (server + backgroundServices):
- Mount the ConfigMap at /etc/prefect/logging
- Set PREFECT_LOGGING_SETTINGS_PATH=/etc/prefect/logging/logging.yml
Depending on your chart version, the keys are typically extraVolumes/extraVolumeMounts. If your chart exposes those, patch like this:
spec:
values:
server:
env:
- name: PREFECT_LOGGING_SETTINGS_PATH
value: /etc/prefect/logging/logging.yml
# keep your existing env here...
extraVolumes:
- name: prefect-logging
configMap:
name: prefect-logging
extraVolumeMounts:
- name: prefect-logging
mountPath: /etc/prefect/logging
readOnly: true
backgroundServices:
env:
- name: PREFECT_LOGGING_SETTINGS_PATH
value: /etc/prefect/logging/logging.yml
# keep your existing env here...
extraVolumes:
- name: prefect-logging
configMap:
name: prefect-logging
extraVolumeMounts:
- name: prefect-logging
mountPath: /etc/prefect/logging
readOnly: true
If your chart uses different keys (e.g., podTemplate, additionalVolumes, additionalVolumeMounts), let me know the chart version and I’ll translate this to the exact fields.
3) Ensure flow-run pods also emit JSON (critical for Datadog)
The HelmRelease only affects the server pods. Your flow and task logs come from flow-run pods spawned by your Kubernetes work pool(s). Update the work pool base job template to mount the same ConfigMap and set the env var:
- Export the default:
prefect work-pool get-default-base-job-template --type kubernetes -f base-template.json
- Edit to add a volume, mount, and env (showing only the additions):
{
"job_configuration": {
"job_manifest": {
"spec": {
"template": {
"spec": {
"volumes": [
{
"name": "prefect-logging",
"configMap": {
"name": "prefect-logging"
}
}
],
"containers": [
{
"name": "prefect-job",
"env": [
{
"name": "PREFECT_LOGGING_SETTINGS_PATH",
"value": "/etc/prefect/logging/logging.yml"
}
],
"volumeMounts": [
{
"name": "prefect-logging",
"mountPath": "/etc/prefect/logging",
"readOnly": true
}
]
}
]
}
}
}
}
}
}
- Apply it:
prefect work-pool update "<your-pool-name>" --base-job-template base-template.json
NotesMarvin
11/17/2025, 4:04 PM