https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I am running prefect through helm on ...
# ask-marvin
h
@Marvin I am running prefect through helm on kubernetes. My helm settings result in one server, 2 worker and 1 prefect-server-background-services deployments. It seems that all of these deployments have a memory leak somewhere as memory utilisation is slowly building up across days which the ultimately kills the pods (OOMkilled) and restarts them, repeating the cycle. Is there any recommendation for these deployments in terms of memory settings?
m
✅ thought for 248.3 seconds
Thanks for the detailed context — that’s frustrating to chase. A few things up front: - There isn’t a single “official” memory spec that fits all Prefect Helm installs; it really depends on run volume, log/event throughput, and whether background services are split out. There also aren’t any currently known, reproducible memory leaks in the latest Prefect 3.x server/worker images, so we’ll want to rule out workload-driven growth. - We can absolutely tune your chart and settings to stabilize things. I’ll suggest safe starting points and a short checklist to isolate the source of growth. Questions to help me tailor exact guidance - Prefect version and Helm chart name/version you’re using - Your values.yaml (or the resource/env sections for server, workers, and background services) - Approx flow runs/day and log volume (e.g., lots of INFO/DEBUG logs?) - Are background services running as a separate deployment? - DB backend (Cloud-hosted vs self-hosted Postgres) and approximate DB size Practical starting points for resources These are conservative baselines that work well for many medium workloads. You can raise/lower once we see your throughput. - Prefect Server (API) - requests: 512Mi–1Gi - limits: 2Gi - Background services (if separate; scheduler, event/log services) - requests: 1–2Gi - limits: 2–4Gi - Kubernetes workers (the long-running process that polls for work — not the job pods that run your flows) - requests: 256–512Mi - limits: 1–2Gi In Helm values.yaml (keys vary slightly by chart, but generally look like this):
Copy code
server:
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "1"
      memory: "2Gi"

backgroundServices:
  runAsSeparateDeployment: true
  resources:
    requests:
      cpu: "500m"
      memory: "2Gi"
    limits:
      cpu: "2"
      memory: "4Gi"

worker:
  resources:
    requests:
      cpu: "250m"
      memory: "512Mi"
    limits:
      cpu: "1"
      memory: "2Gi"
If your chart nests these sections differently, share your values.yaml and I’ll map them correctly. Immediate mitigations to stop the OOM cycle while we diagnose - Temporarily increase memory limits (especially for background services) to keep pods stable while we test. - Reduce logs sent to the API to see if memory stabilizes: - Set env 
PREFECT_LOGGING_TO_API=false
(and/or raise 
PREFECT_LOGGING_LEVEL=WARNING
) on workers and/or server to cut log volume. This is a quick way to check if log batching/ingestion is the driver. - If you’re self-hosting Postgres, ensure vacuum/analyze is running regularly; very large tables (especially events/logs/runs) can cause server-side memory pressure when background services query/process them. What typically drives steady memory growth over days - Background services handling lots of events/logs/scheduling work (especially when separated, they’ll show the growth more clearly) - Very verbose logs sent to the API (workers and flows producing lots of logs that the server ingests/persists) - Database size/maintenance (heavy churn on runs/logs/events over time without adequate vacuum can amplify memory use during queries/batching) How we can isolate the culprit quickly - Compare memory curves by deployment: - If server and background services both grow, start by reducing logs-to-API and checking DB size. - If workers grow but server is steady, try disabling logs-to-API just on workers and watch memory for 24–48h. - Share a 24–72h graph of memory usage per deployment and rough run/log volume so we can correlate. - Confirm your Prefect and Helm chart versions; upgrading to the latest 3.x often includes stability improvements. Next steps - If you share your Helm chart version and values.yaml (redact secrets), I’ll: - Verify the correct resource keys for your chart - Suggest specific env toggles (and where to put them) to reduce log/event pressure - Recommend HPA settings if appropriate
If you prefer, I can also provide a minimal values.yaml you can drop in to: - Split background services - Set the resource requests/limits above - Turn off logs-to-API for a 2–3 day test window Once we see which component is actually growing and how fast, we can decide whether to tune retention/logging further or bump resources permanently.
h
@Marvin these are my deployments + images used. The second screenshot shows the helm app deployments and their versions. What else do you need
m
✅ thought for 175.4 seconds
Got it — thanks! To zero in on the cause and give precise settings, I’ll need a few specifics you can paste here. The more of these you can share, the faster we can pinpoint the memory driver: What to paste - Helm/chart details - Chart name/repo and version - Output of:
Copy code
helm get values <release-name> -n <namespace>
or at least the relevant sections for: - 
server.resources
, 
server.env
- 
backgroundServices.runAsSeparateDeployment
, 
backgroundServices.resources
, 
backgroundServices.env
- 
worker.resources
, 
worker.env
- Images/tags actually running - The screenshot helps; a text paste is even better (redact registry if needed). For example:
Copy code
prefecthq/prefect:<tag>  # for server
    prefecthq/prefect:<tag>  # for background services (if separate)
    prefecthq/prefect:<tag>  # for worker
- Which pods are growing - Over 24–72h, which of these show monotonic memory growth: server, background-services, worker? - Any HPA on these deployments? - Recent OOMKilled events from:
Copy code
kubectl describe pod <pod-name> -n <namespace>
- Workload characteristics - Approx runs/day and peak concurrency - Log verbosity (INFO/DEBUG?) and whether you’re sending logs to the API - Any high-volume event usage (automations, event-driven triggers) - Database - Postgres version and where it’s hosted (cloud-managed vs self-managed) - Rough DB size and top tables (if you can, share the result of):
Copy code
-- Top 5 largest tables
    SELECT relname AS table, pg_size_pretty(pg_total_relation_size(relid)) AS size
    FROM pg_catalog.pg_statio_user_tables
    ORDER BY pg_total_relation_size(relid) DESC
    LIMIT 5;
- Any custom retention settings you’ve applied Quick checks you can run now (optional, helps triage) - A 24–48h test lowering log volume from flows/workers (e.g., set logging to WARNING) to see if pod memory stabilizes - If background services are separate, watch just that deployment’s memory — it often highlights events/log ingestion pressure References - Prefect Helm charts - Prefect + Kubernetes deployment guide Once I have the values/env and which pod is growing, I’ll propose exact resource requests/limits and any env toggles that are appropriate for your setup.
h
@Marvin helm server > prefect-server:2025.11.18162334 helm worker > prefect-worker:2025.11.14001416 helm server deployment yaml:
Copy code
backgroundServices:
  ...
  resources:
    limits:
      cpu: '1'
      memory: 1Gi
    requests:
      cpu: 500m
      memory: 512Mi
  revisionHistoryLimit: 10
  runAsSeparateDeployment: true
  ...
global:
  prefect:
    env: []
    image:
      prefectTag: 3.6.2-python3.11
      pullPolicy: IfNotPresent
      pullSecrets: []
      repository: prefecthq/prefect
  cattle:
    systemProjectId: p-h2hmp
httproute:
  annotations: {}
  enabled: true
  extraRules: []
  hostnames: []
  labels: {}
  name: ''
  parentRefs:
    - name: ''
      namespace: ''
      port: null
      sectionName: https
  path: /
  tls:
    redirect: false
    redirectPort: 443
ingress:
  annotations: {}
  className: ''
  enabled: false
  extraHosts: []
  extraPaths: []
  extraRules: []
  extraTls: []
  host:
    hostname: prefect.local
    path: /
    pathType: ImplementationSpecific
  selfSigned: false
  servicePort: server-svc-port
  tls: false
migrations:
  affinity: {}
  backoffLimit: 5
  command: |
    prefect server database upgrade -y
  enabled: true
  entrypoint:
    - /bin/sh
    - '-c'
  env: []
  extraVolumeMounts: []
  extraVolumes: []
  nodeSelector:
    prefect: enabled
  resources:
    limits:
      cpu: 500m
      memory: 256Mi
    requests:
      cpu: 100m
      memory: 128Mi
  restartPolicy: Never
  securityContext:
    allowPrivilegeEscalation: false
    capabilities: {}
    readOnlyRootFilesystem: true
    runAsNonRoot: true
    runAsUser: 1001
  timeoutSeconds: 300
  tolerations:
    - effect: NoSchedule
      key: bm-worker
      operator: Exists
nameOverride: ''
namespaceOverride: ''
postgresql:
  auth:
    database: null
    enablePostgresUser: null
    password: null
    username: null
    existingSecret: prefect-server
  enabled: false
  image:
    repository: null
    tag: null
  primary:
    initdb:
      user: null
    persistence:
      enabled: null
redis:
  architecture: null
  enabled: false
  image:
    repository: null
    tag: null
secret:
  create: null
  database: null
  host: null
  name: null
  password: null
  port: null
  username: null
server:
  affinity: {}
  apiBasePath: /api
  args: []
  autoscaling:
    enabled: false
    maxReplicas: 100
    minReplicas: 1
    targetCPU: 80
    targetMemory: 80
  basicAuth:
    authString: null
    enabled: true
    existingSecret: prefect-server
  command: []
  containerSecurityContext:
    allowPrivilegeEscalation: false
    capabilities: {}
    readOnlyRootFilesystem: true
    runAsNonRoot: true
    runAsUser: 1001
  debug: false
  env:
    - name: PREFECT_EVENTS_RETENTION_PERIOD
      value: 3d
    - name: PREFECT_API_DATABASE_TIMEOUT
      value: '1800'
  extraArgs: []
  extraContainers: []
  extraEnvVarsCM: ''
  extraEnvVarsSecret: ''
  extraVolumeMounts: []
  extraVolumes: []
  livenessProbe:
    config:
      failureThreshold: 3
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    enabled: false
  loggingLevel: WARNING
  nodeSelector:
    prefect: enabled
  podAnnotations: {}
  podLabels: {}
  podSecurityContext:
    fsGroup: 1001
    runAsNonRoot: true
    runAsUser: 1001
    seccompProfile:
      type: RuntimeDefault
  priorityClassName: ''
  readinessProbe:
    config:
      failureThreshold: 3
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    enabled: false
  replicaCount: 1
  resources:
    limits:
      cpu: '1'
      memory: 1Gi
    requests:
      cpu: 500m
      memory: 512Mi
  revisionHistoryLimit: 10
  tolerations:
    - effect: NoSchedule
      key: bm-worker
      operator: Exists
  uiConfig:
    prefectUiApiUrl: <http://localhost:4200/api>
    prefectUiStaticDirectory: /ui_build
  updateStrategy:
    type: RollingUpdate
service:
  annotations: {}
  clusterIP: ''
  externalTrafficPolicy: Cluster
  extraPorts: []
  nodePort: ''
  port: 4200
  targetPort: 4200
  type: ClusterIP
serviceAccount:
  annotations: {}
  create: true
  name: ''
sqlite:
  enabled: false
  persistence:
    enabled: null
    size: null
    storageClassName: null
m
Your message was too long, here's your message at the allowed limit: helm server > prefect-server 2025 11 18162334 helm worker > prefect-worker 2025 11 14001416 helm server deployment yaml ```backgroundServices affinity {} args [] command [] containerSecurityContext allowPrivilegeEscalation false capabilities {} readOnlyRootFilesystem true runAsNonRoot true runAsUser 1001 debug false env [] extraContainers [] extraEnvVarsCM '' extraEnvVarsSecret '' extraVolumeMounts [] extraVolumes [] loggingLevel WARNING messaging broker prefect_redis messaging cache prefect_redis messaging redis db 0 host prefect-redis prefect svc cluster local password '' port 6379 ssl false username '' nodeSelector prefect enabled podAnnotations {} podLabels {} podSecurityContext fsGroup 1001 runAsNonRoot true runAsUser 1001 priorityClassName '' resources limits cpu '1' memory 1Gi requests cpu 500m memory 512Mi revisionHistoryLimit 10 runAsSeparateDeployment true serviceAccount annotations {} create true name '' tolerations - effect NoSchedule key bm-worker operator Exists commonAnnotations {} commonLabels {} fullnameOverride prefect-server gateway annotations {} className '' enabled false infrastructure {} labels {} listeners - hostname '' name http port 80 protocol HTTP - hostname '' name https port 443 protocol HTTPS tls certificateRefs - kind Secret name '' namespace '' mode Terminate name '' global prefect env [] image prefectTag 3 6 2-python3 11 pullPolicy IfNotPresent pullSecrets [] repository prefecthq/prefect cattle systemProjectId p-h2hmp httproute annotations {} enabled true extraRules [] hostnames [] labels {} name '' parentRefs - name '' namespace '' port null sectionName https path / tls redirect false redirectPort 443 ingress annotations {} className '' enabled false extraHosts [] extraPaths [] extraRules [] extraTls [] host hostname prefect local path / pathType ImplementationSpecific selfSigned false servicePort server-svc-port tls false migrations affinity {} backoffLimit 5 command | prefect server database upgrade -y enabled true entrypoint - /bin/sh - '-c' env [] extraVolumeMounts [] extraVolumes [] nodeSelector prefect enabled resources limits cpu 500m memory 256Mi requests cpu 100m memory 128Mi restartPolicy Never securityContext allowPrivilegeEscalation false capabilities {} readOnlyRootFilesystem true runAsNonRoot true runAsUser 1001 timeoutSeconds 300 tolerations - effect NoSchedule key bm-worker operator Exists nameOverride '' namespaceOverride '' postgresql auth database null enablePostgresUser null password null username null existingSecret prefect-server enabled false image repository null tag null primary initdb user null persistence enabled null redis architecture null enabled false image repository null tag null secret create null database null host null name null password null port null username null server affinity {} apiBasePath /api args [] autoscaling enabled false maxReplicas 100 minReplicas 1 targetCPU 80 targetMemory 80 basicAuth authString null enabled true existingSecret prefect-server command [] containerSecurityContext allowPrivilegeEscalation false capabilities {} readOnlyRootFilesystem true runAsNonRoot true runAsUser 1001 debug false env - name PREFECT_EVENTS_RETENTION_PERIOD value 3d - name PREFECT_API_DATABASE_TIMEOUT value '1800' extraArgs [] extraContainers [] extraEnvVarsCM '' extraEnvVarsSecret '' extraVolumeMounts [] extraVolumes [] livenessProbe config failureThreshold 3 initialDelaySeconds 10 periodSeconds 10 successThreshold 1 timeoutSeconds 5 enabled false loggingLevel WARNING nodeSelector prefect enabled podAnnotations {} podLabels {} podSecurityContext fsGroup 1001 runAsNonRoot true runAsUser 1001 seccompProfile type RuntimeDefault priorityClassName '' readinessProbe config failureThreshold 3 initialDelaySeconds 10 periodSeconds 10 successThreshold 1 timeoutSeconds 5 enabled false replicaCount 1 resources limits cpu '1' memory 1Gi requests cpu 500m memory 512Mi revisionHistoryLimit 10 tolera
tions - effect NoSchedule key bm-worker operator
Your message was too long, here's your message at the allowed limit: helm server > prefect-server 2025 11 18162334 helm worker > prefect-worker 2025 11 14001416 helm server deployment yaml ```backgroundServices affinity {} args [] command [] containerSecurityContext allowPrivilegeEscalation false capabilities {} readOnlyRootFilesystem true runAsNonRoot true runAsUser 1001 debug false env [] extraContainers [] extraEnvVarsCM '' extraEnvVarsSecret '' extraVolumeMounts [] extraVolumes [] loggingLevel WARNING messaging broker prefect_redis messaging cache prefect_redis messaging redis db 0 host prefect-redis prefect svc cluster local password '' port 6379 ssl false username '' nodeSelector prefect enabled podAnnotations {} podLabels {} podSecurityContext fsGroup 1001 runAsNonRoot true runAsUser 1001 priorityClassName '' resources limits cpu '1' memory 1Gi requests cpu 500m memory 512Mi revisionHistoryLimit 10 runAsSeparateDeployment true serviceAccount annotations {} create true name '' tolerations - effect NoSchedule key bm-worker operator Exists commonAnnotations {} commonLabels {} fullnameOverride prefect-server gateway annotations {} className '' enabled false infrastructure {} labels {} listeners - hostname '' name http port 80 protocol HTTP - hostname '' name https port 443 protocol HTTPS tls certificateRefs - kind Secret name '' namespace '' mode Terminate name '' global prefect env [] image prefectTag 3 6 2-python3 11 pullPolicy IfNotPresent pullSecrets [] repository prefecthq/prefect cattle systemProjectId p-h2hmp httproute annotations {} enabled true extraRules [] hostnames [] labels {} name '' parentRefs - name '' namespace '' port null sectionName https path / tls redirect false redirectPort 443 ingress annotations {} className '' enabled false extraHosts [] extraPaths [] extraRules [] extraTls [] host hostname prefect local path / pathType ImplementationSpecific selfSigned false servicePort server-svc-port tls false migrations affinity {} backoffLimit 5 command | prefect server database upgrade -y enabled true entrypoint - /bin/sh - '-c' env [] extraVolumeMounts [] extraVolumes [] nodeSelector prefect enabled resources limits cpu 500m memory 256Mi requests cpu 100m memory 128Mi restartPolicy Never securityContext allowPrivilegeEscalation false capabilities {} readOnlyRootFilesystem true runAsNonRoot true runAsUser 1001 timeoutSeconds 300 tolerations - effect NoSchedule key bm-worker operator Exists nameOverride '' namespaceOverride '' postgresql auth database null enablePostgresUser null password null username null existingSecret prefect-server enabled false image repository null tag null primary initdb user null persistence enabled null redis architecture null enabled false image repository null tag null secret create null database null host null name null password null port null username null server affinity {} apiBasePath /api args [] autoscaling enabled false maxReplicas 100 minReplicas 1 targetCPU 80 targetMemory 80 basicAuth authString null enabled true existingSecret prefect-server command [] containerSecurityContext allowPrivilegeEscalation false capabilities {} readOnlyRootFilesystem true runAsNonRoot true runAsUser 1001 debug false env - name PREFECT_EVENTS_RETENTION_PERIOD value 3d - name PREFECT_API_DATABASE_TIMEOUT value '1800' extraArgs [] extraContainers [] extraEnvVarsCM '' extraEnvVarsSecret '' extraVolumeMounts [] extraVolumes [] livenessProbe config failureThreshold 3 initialDelaySeconds 10 periodSeconds 10 successThreshold 1 timeoutSeconds 5 enabled false loggingLevel WARNING nodeSelector prefect enabled podAnnotations {} podLabels {} podSecurityContext fsGroup 1001 runAsNonRoot true runAsUser 1001 seccompProfile type RuntimeDefault priorityClassName '' readinessProbe config failureThreshold 3 initialDelaySeconds 10 periodSeconds 10 successThreshold 1 timeoutSeconds 5 enabled false replicaCount 1 resources limits cpu '1' memory 1Gi requests cpu 500m memory 512Mi revisionHistoryLimit 10 tolerations
- effect NoSchedule key bm-worker operator
✅ thought for 65.5 seconds
Awesome, thanks — this helps a lot. A few quick observations from your values: - You’re on Prefect 3.6.2 (image tag 3.6.2-python3.11) with background services split out and both server/background-services capped at 1Gi. In most real workloads, 1Gi for background services is tight and tends to creep up over days as events/logs/scheduling work accumulate. - You’ve set PREFECT_EVENTS_RETENTION_PERIOD=3d (good!) but the in-memory event persister queue isn’t capped explicitly, so background services can still hold quite a bit in memory between flushes under load. What I still need - The worker deployment’s values (resources + env) so I can recommend exact changes there too. - Which pod(s) are OOMKilled first and the rough runs/day + log volume (INFO vs WARNING/DEBUG). Immediate changes I recommend (safe, targeted) 1) Raise memory on server and especially background services
Copy code
server:
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "1"
      memory: "2Gi"

backgroundServices:
  runAsSeparateDeployment: true
  resources:
    requests:
      cpu: "500m"
      memory: "2Gi"
    limits:
      cpu: "2"
      memory: "4Gi"
For workers (please share your current values), a good starting point is:
Copy code
worker:
  resources:
    requests:
      cpu: "250m"
      memory: "512Mi"
    limits:
      cpu: "1"
      memory: "2Gi"
2) Cap the in-memory event persister queue Add to the server (and background services if you set env separately on that deployment):
Copy code
server:
  env:
    - name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE
      value: "20000"        # default is higher; 20k is safer in constrained clusters
    - name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE
      value: "20"           # keep defaults unless you see backpressure
    - name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL
      value: "5"
This prevents unbounded in-memory growth when event volume spikes. 3) Keep event retention short while tuning You already have:
Copy code
- name: PREFECT_EVENTS_RETENTION_PERIOD
  value: 3d
That’s a good start. If your throughput is very high, try 1–2d temporarily while we stabilize memory. 4) Reduce log pressure from workers/flows (quick isolation step) - Temporarily raise logging to WARNING to see if memory stabilizes:
Copy code
worker:
  env:
    - name: PREFECT_LOGGING_LEVEL
      value: WARNING
If you’re sending a lot of logs to the API, this commonly shows up as slow-but-steady growth in background services. 5) Check Postgres table sizes Large events/logs/run tables amplify memory during background processing. If you can, run:
Copy code
-- Top 5 largest tables
SELECT relname AS table, pg_size_pretty(pg_total_relation_size(relid)) AS size
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC
LIMIT 5;
If events/logs are dominating, retention + the queue cap above almost always helps. Optional: upgrade to latest 3.x 3.6.2 is solid, but if you can bump to the latest 3.x patch you’ll pick up stability fixes. See Prefect releases. If you share your worker values (and which pod grows the fastest), I’ll tailor the exact env/resource changes for that deployment and suggest any additional toggles specific to your workload. References - Helm charts: Prefect Helm charts - Settings reference (env vars): Prefect settings
h
Copy code
commonAnnotations: {}
commonLabels: {}
fullnameOverride: prefect-worker
nameOverride: ''
namespaceOverride: ''
role:
  create: true
  extraPermissions:
    - apiGroups:
        - ''
      resources:
        - pods
        - pods/log
        - pods/status
      verbs:
        - get
        - watch
        - list
    - apiGroups:
        - batch
      resources:
        - jobs
        - events
      verbs:
        - get
        - list
        - watch
        - create
        - update
        - patch
        - delete
  namespace: null
rolebinding:
  create: true
serviceAccount:
  annotations: {}
  create: true
  name: ''
worker:
  affinity: {}
  apiConfig: selfHostedServer
  args: []
  autoscaling:
    enabled: false
    maxReplicas: 1
    minReplicas: 1
    targetCPUUtilizationPercentage: 80
    targetMemoryUtilizationPercentage: 80
  cloudApiConfig:
    accountId: ''
    apiKeySecret:
      key: key
      name: prefect-api-key
    cloudUrl: <https://api.prefect.cloud/api>
    workspaceId: ''
  clusterUid: ''
  command: []
  config:
    baseJobTemplate:
      configuration: null
      existingConfigMapName: base-job-template
    http2: true
    installPolicy: prompt
    jobNamespace: null
    limit: null
    name: null
    prefetchSeconds: 10
    queryInterval: 5
    type: kubernetes
    workPool: prod-worker-1
    workQueues: []
  containerSecurityContext:
    allowPrivilegeEscalation: false
    capabilities: {}
    readOnlyRootFilesystem: true
    runAsNonRoot: true
    runAsUser: 1001
  dnsConfig:
    nameservers: []
    options: []
    searches: []
  dnsPolicy: ''
  extraArgs: []
  extraContainers: []
  extraEnvVars: []
  extraEnvVarsCM: ''
  extraEnvVarsCMs: []
  extraEnvVarsSecret: ''
  extraEnvVarsSecrets: []
  extraVolumeMounts: []
  extraVolumes: []
  image:
    debug: false
    prefectTag: 3.6.2-python3.11-kubernetes
    pullPolicy: IfNotPresent
    pullSecrets: []
    repository: prefecthq/prefect
  initContainer:
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities: {}
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1001
    extraContainers: []
    resources: {}
  livenessProbe:
    config:
      failureThreshold: 3
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    enabled: false
  nodeSelector:
    prefect: enabled
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  podSecurityContext:
    fsGroup: 1001
    runAsNonRoot: true
    runAsUser: 1001
    seccompProfile:
      type: RuntimeDefault
  priorityClassName: ''
  replicaCount: 1
  resources:
    limits:
      cpu: 1000m
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 256Mi
  revisionHistoryLimit: 10
  selfHostedServerApiConfig:
    apiUrl: <http://prefect-server.prefect.svc.cluster.local:4200/api>
    basicAuth:
      authString: null
      enabled: true
      existingSecret: prefect-server
  selfManagedCloudApiConfig:
    accountId: ''
    apiKeySecret:
      key: key
      name: prefect-api-key
    apiUrl: ''
    cloudApiUrl: ''
    workspaceId: ''
  tolerations:
    - effect: NoSchedule
      key: bm-worker
      operator: Exists
global:
  cattle:
    systemProjectId: p-h2hmp
this is my prefect worker deployment yaml @Marvin
m
✅ thought for 313.6 seconds
Thanks — this is perfect. A few concrete, chart-specific changes I recommend based on your values: Key findings from your configs - Server, background services, and worker are all capped at 1Gi memory. With background services split out and handling events/logs/scheduling, 1Gi is usually too tight and will creep upward over days. - You set PREFECT_EVENTS_RETENTION_PERIOD=3d under server.env only. Since background services run in a separate deployment, they won’t see that env var unless you also set it on the background services pod (or via global env). Targeted changes to stabilize memory 1) Raise memory limits (especially background services)
Copy code
server:
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "1"
      memory: "2Gi"

backgroundServices:
  runAsSeparateDeployment: true
  resources:
    requests:
      cpu: "500m"
      memory: "2Gi"
    limits:
      cpu: "2"
      memory: "4Gi"
2) Apply retention and cap the in-memory events queue on background services Add these env vars to background services so they actually take effect there:
Copy code
backgroundServices:
  env:
    - name: PREFECT_EVENTS_RETENTION_PERIOD
      value: "3d"     # move/copy from server.env so background services see it
    - name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE
      value: "20000"  # cap the in-memory buffer to prevent slow creep
    # Optional fine-tuning (usually not needed unless you see backpressure):
    # - name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE
    #   value: "20"
    # - name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL
    #   value: "5"
If your chart doesn’t expose backgroundServices.env, you can place these under global.prefect.env to apply cluster-wide, or let me know and I’ll map to your chart’s exact key. 3) Temporarily reduce log pressure from the worker (quick isolation step) By default, workers send logs to the API at INFO level. To see if logs are the driver, try:
Copy code
worker:
  extraEnvVars:
    - name: PREFECT_LOGGING_TO_API_ENABLED
      value: "false"
    - name: PREFECT_LOGGING_LEVEL
      value: "WARNING"
If memory stabilizes over 24–48h, we can re-enable logs-to-API with smaller batches:
Copy code
- name: PREFECT_LOGGING_TO_API_ENABLED
      value: "true"
    - name: PREFECT_LOGGING_TO_API_BATCH_SIZE
      value: "2000000"
    - name: PREFECT_LOGGING_TO_API_BATCH_INTERVAL
      value: "5.0"
4) Worker resources Your worker is requests: 256Mi, limits: 1Gi. That’s fine as a starting point; if you see the worker pod creeping, bump to 512Mi/2Gi:
Copy code
worker:
  resources:
    requests:
      cpu: "250m"
      memory: "512Mi"
    limits:
      cpu: "1"
      memory: "2Gi"
Why these changes help - Background services do the heavy lifting for events/log ingestion and scheduling—moving retention to that pod and capping the persister queue prevents unbounded in-memory buffers. - Disabling or reducing logs-to-API is a fast way to test if log batching/ingestion is the memory driver. - Raising limits stops the OOM cycle while we observe the effect of the queue cap and logging adjustments. What I still need (to tune further) - Which deployment OOMKills first: server, background-services, or worker? - Approx runs/day and whether flows are verbose (INFO/DEBUG) with lots of log lines. - If possible, a quick check of top Postgres tables (events/logs/runs) size — large tables amplify background-service memory during processing:
Copy code
SELECT relname AS table, pg_size_pretty(pg_total_relation_size(relid)) AS size
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC
LIMIT 5;
If you apply the snippets above and share which pod was growing before/after, I’ll help dial in the final resource and env settings. Refs - Helm charts: Prefect Helm charts - Settings (all env vars): Prefect settings
24 Views
Open in Slack
PreviousNext
Powered by