Thanks for flagging this — kernel slab growth leading to OOM after a week usually points to OS-level caching/leaks from high churn in files/sockets/containers rather than a pure Python heap issue. We can triage both the Prefect side and the host/kernel side. A couple of quick questions to narrow it down: - Prefect version and worker type: process, docker, kubernetes, or something custom? - Where is the worker running: bare VM, Docker, or Kubernetes? Which OS/kernel version? - Typical workload: lots of short runs vs a few long runs? Rough runs/day and concurrency? - Do your flows create many temp files or touch large directories (e.g., unpack archives, write logs locally, create many small files)? - Which slab caches are the largest when it crashes (dentry, inode_cache, kmalloc-*, skbuff, sock, overlayfs entries, nf_conntrack, etc.)? Short-term mitigations - Upgrade Prefect to the latest 3.x. There have been fixes around long-lived connections/websockets and stability. - Reduce worker background churn by adjusting polling/heartbeat:

# pick values appropriate for your environment
  export PREFECT_WORKER_QUERY_SECONDS=15
  export PREFECT_WORKER_HEARTBEAT_SECONDS=60
  export PREFECT_WORKER_PREFETCH_SECONDS=5

Docs: Work pools & workers - Cap concurrency on the worker to reduce load (i.e., total simultaneous flow runs). - Schedule a periodic worker restart (systemd/K8s restart policy or a weekly cron) as a stopgap to prevent week-long accumulation. - If you’re on Docker/K8s: - Ensure job containers are removed after runs and old images/volumes are pruned regularly. - Enable log rotation and prune old container logs to prevent inode/dentry churn. - Consider memory limits on the worker container so an OOM kills just the worker and it’s restarted cleanly. How to pinpoint the slab culprit Run these on the host when memory is high: - Which slabs are big?

slabtop -o
  # or
  cat /proc/slabinfo | sort -k3 -nr | head

- Large dentry/inode_cache: tons of filesystem churn (many files/dirs). Check for flows writing to /tmp or big trees; add cleanup/rotation; consider remote storage for results/artifacts; schedule tmp cleanup. - Large skbuff/sock/tcp-related slabs: many sockets/connection churn. Check:

ss -s
    netstat -anp | grep TIME_WAIT | wc -l

If high, look for libraries in your flows creating new clients per call; reuse HTTP clients where possible. - Large overlayfs-related slabs (overlay_inode, etc.): Docker overlay pressure; prune containers/images/volumes; reduce image churn; ensure containers truly exit and are removed after each run. - Large nf_conntrack: NAT table growth; tune conntrack max and GC, or reduce outbound connection fan-out. - Check open fds of the worker:

pidof -s python  # or pgrep -f "prefect worker"
  ls -l /proc/<PID>/fd | wc -l

If growing over time, something is holding descriptors (files/sockets) open. - Inspect kernel reclaimables:

grep -E 'Slab|SReclaimable|SUnreclaim' /proc/meminfo

High SReclaimable with pressure suggests caches will drop if reclaim works; if they don’t, something is pinning entries. Temporary kernel-side relief (use with caution) - One-time cache drop (root):

echo 2 > /proc/sys/vm/drop_caches   # dentries/inodes
  # or
  echo 3 > /proc/sys/vm/drop_caches   # pagecache + dentries/inodes

- Consider slightly higher reclaim pressure:

sysctl -w vm.vfs_cache_pressure=200

Test carefully; this trades cache performance for memory reclaim. Prefect-side knobs that can help reduce churn - Confirm you’re on Prefect 3.x and latest:

prefect version
  pip install -U prefect

- Worker background settings (assumes 3.x): -

PREFECT_WORKER_QUERY_SECONDS

(default 10) -

PREFECT_WORKER_HEARTBEAT_SECONDS

(default 30) -

PREFECT_WORKER_PREFETCH_SECONDS

(default 10)

-

PREFECT_WORKER_CANCELLATION_POLL_SECONDS

(fallback when cancellation websockets are off) - Reduce API/log chatter if your flows are very chatty with logs. Batch or lower log level where feasible. - If your flows write to local disk, add explicit cleanup at the end of runs, and consider directing temp output to a dedicated path with periodic cleanup. Next steps If you can share: - The top 10 slab caches from

slabtop -o

when it’s near-OOM - Prefect version, worker type, host environment (OS/kernel, Docker/K8s versions) - Rough run volume/concurrency and whether the workload writes lots of files or makes many outbound HTTP calls …I can help pinpoint root cause and recommend precise fixes (e.g., Docker/K8s worker config, flow code patterns, or OS tuning). If this boils down to a reproducible Prefect worker leak, I’ll open an issue/discussion with full details.

Thanks — that context helps a lot. Given: - Prefect 3.x - Process worker running in Kubernetes - ~100k short runs/day - Kernel slab growth dominated by dentry This pattern is classic VFS/overlayfs churn: each short run creates and deletes a temp working dir. With process workers, that happens inside the same long-lived pod/container; on many Kubernetes/node/kernel combos the upper overlayfs layer holds onto dentries longer than expected, so dentry/inode slabs balloon over time. What to do next (in order of impact) 1) Move temp/working dirs off the container overlayfs - Mount an emptyDir volume inside the worker pod for /tmp so every run’s TemporaryDirectory is on emptyDir instead of overlayfs. - Optionally set emptyDir medium: Memory (tmpfs) to bypass overlay entirely and confine the cache to pod memory; add a sizeLimit to avoid surprises. Example Deployment snippet:

spec:
  template:
    spec:
      containers:
      - name: prefect-worker
        image: your-image
        # Ensure Python temp uses /tmp (default), or explicitly:
        env:
        - name: TMPDIR
          value: /tmp
        volumeMounts:
        - name: tmp
          mountPath: /tmp
      volumes:
      - name: tmp
        emptyDir:
          # medium: Memory      # uncomment to use tmpfs
          # sizeLimit: 2Gi      # optional guardrail

Alternative: use a dedicated working dir - Mount an emptyDir (disk or tmpfs) at /work and point the process worker’s working_dir there. That avoids touching overlayfs even if TMPDIR changes later. Update the process work pool base job template to set working_dir and TMPDIR:

prefect work-pool get-default-base-job-template --type process > process.json
# Edit process.json to include:
# "job_configuration": { "working_dir": "/work", "env": { "TMPDIR": "/work/tmp" }, ... }
# "variables": { ... ensure working_dir default and env include TMPDIR ... }
prefect work-pool update "your-pool" --base-job-template process.json

Docs: Work pools & workers 2) Distribute load across pods/nodes - Run multiple worker replicas and use a work-queue per replica or shared queues with per-worker limits. This reduces per-node filesystem churn and slab pressure. - Set a sensible

--limit

on each worker to cap concurrent runs. 3) Keep background churn modest With this many short runs, the dentry issue dominates, but it still helps to reduce idle churn:

export PREFECT_WORKER_QUERY_SECONDS=15
export PREFECT_WORKER_HEARTBEAT_SECONDS=60
export PREFECT_WORKER_PREFETCH_SECONDS=5

4) Node-level mitigations (if slab remains high) - Prefer newer kernels (there have been overlayfs dentry/inode reclaim fixes over time). - Increase VFS cache pressure to encourage reclaim:

sysctl -w vm.vfs_cache_pressure=200

- As a stopgap, cron a safe-time window to drop caches:

echo 2 > /proc/sys/vm/drop_caches

Use with care in production. 5) Flow code hygiene - Ensure flows don’t create excessive small files or leave descriptors open; always use context managers for files and sockets. - Avoid writing to the container filesystem; prefer mounted emptyDir paths or remote object storage for artifacts. Why this helps - Process workers create a TemporaryDirectory per run; at your volume, that’s a constant stream of create/unlink. On overlayfs-backed container filesystems, dentries can be retained longer, growing slab. Routing temp and working dirs to emptyDir (preferably tmpfs) avoids overlayfs’s upper layer and makes reclaim more predictable. emptyDir also gets torn down when pods roll, preventing week-long accumulation. If you can, after mounting emptyDir for /tmp (or /work + TMPDIR), please watch: - slabtop -o (dentry/inode after a day) - Pod memory usage (if using tmpfs + sizeLimit) - Worker replicas per node If slab still climbs abnormally, share: - Kernel and container runtime versions - A slabtop -o top 10 excerpt after a day

- Your worker Deployment YAML volume mounts and env …and I’ll help fine-tune further (or we’ll escalate as a reproducible bug if needed).

Got it — lots of short runs with a process worker in Kubernetes and dentry at the top points straight at per-run temp/working-dir churn on the container filesystem (overlayfs). Since you haven’t set a working directory, the process worker creates a new TemporaryDirectory for every run under /tmp, then deletes it — great for isolation, but it can balloon dentry/inode slabs on some kernels over time. Two solid ways to fix it: Option A (recommended first): keep per-run isolation, move temp dirs off overlayfs - Mount an emptyDir at /tmp in the worker pod so Python’s TemporaryDirectory uses that volume (not overlayfs). - Optionally use tmpfs (Memory) with a sizeLimit so it’s fully in RAM and promptly reclaimed. Kubernetes Deployment snippet:

spec:
  template:
    spec:
      containers:
      - name: prefect-worker
        image: your-image
        volumeMounts:
        - name: tmp
          mountPath: /tmp
      volumes:
      - name: tmp
        emptyDir:
          # medium: Memory      # uncomment to use tmpfs
          # sizeLimit: 2Gi      # optional guardrail

No Prefect changes needed; each flow run still gets its own cleaned-up temp dir, just on emptyDir instead of overlayfs. Option B: reduce churn by using a stable working directory on emptyDir - Mount an emptyDir at /work and point the process worker’s working_dir there. - Also set TMPDIR to a subfolder (e.g., /work/tmp). Create it with an initContainer. Kubernetes snippet:

spec:
  template:
    spec:
      initContainers:
      - name: init-tmp
        image: busybox
        command: ["sh", "-c", "mkdir -p /work/tmp && chmod 1777 /work/tmp"]
        volumeMounts:
        - name: work
          mountPath: /work
      containers:
      - name: prefect-worker
        image: your-image
        volumeMounts:
        - name: work
          mountPath: /work
      volumes:
      - name: work
        emptyDir: {}

Then set the process work pool base job template to use /work and TMPDIR:

prefect work-pool get-default-base-job-template --type process > process.json
# Edit process.json:
# job_configuration.working_dir = "/work"
# job_configuration.env = { "TMPDIR": "/work/tmp" }  (merge with existing env if present)
prefect work-pool update "YOUR_POOL_NAME" --base-job-template process.json

Trade-offs: - A: Keeps per-run isolation and cleanup; just moves temp dirs to an overlay-free volume. Usually enough to stop dentry growth. - B: Further reduces create/delete churn (no per-run dir creation). Only use if your flows don’t assume a fresh working directory; be careful to avoid cross-run artifacts. Additional tweaks that help at your scale - Spread load across more worker replicas/nodes and set a per-worker concurrency limit. - Gentle background tuning to reduce idle churn:

export PREFECT_WORKER_QUERY_SECONDS=15
  export PREFECT_WORKER_HEARTBEAT_SECONDS=60
  export PREFECT_WORKER_PREFETCH_SECONDS=5

- As a stopgap, have your orchestrator (Deployment/DaemonSet) roll pods weekly to ensure emptyDir reset while you validate. How to verify improvement - Watch dentry/inode after the change:

slabtop -o
  grep -E 'dentry|inode' /proc/slabinfo | sort -k3 -nr | head

- If using tmpfs for /tmp, also keep an eye on pod memory and consider sizeLimit. Docs for reference - Work pools & workers If you can try Option A (emptyDir at /tmp) and share a slabtop -o snapshot a few hours later, I can help confirm it’s addressed and decide if Option B is worth adding.

Short answer: after moving the worker’s temp/working dir to an emptyDir, you should not see a monotonic, unbounded slab (dentry) increase per run. Small, transient bumps are normal, but the kernel should reclaim them and the overall slab usage should plateau instead of climbing indefinitely. What’s expected - Per run, creating/deleting temp dirs/files will momentarily increase dentries/inodes. - With emptyDir (especially tmpfs + sizeLimit), those caches become reclaimable quickly; they may linger while memory is plentiful but should drop under pressure. You should see fluctuations, not steady growth. If you still see steady growth, check these first 1) Verify /tmp is really your emptyDir inside the worker pod:

mount | grep ' /tmp '
   # or
   cat /proc/mounts | grep ' /tmp '
   df -T /tmp

You should see a kubernetes emptyDir mount (or tmpfs if you set medium: Memory). 2) Verify the flow subprocess actually uses that TMPDIR: - Add a tiny test flow that prints

os.getenv("TMPDIR")

and

Path.cwd()

so you know the run inherits the settings from the work pool job template. 3) If you set a fixed working_dir (Option B), ensure runs are not leaving artifacts there. - A shared working_dir that accumulates files will legitimately grow dentries over time. Either keep using per-run /tmp isolation (Option A) or add explicit cleanup. 4) Make sure results/logs aren’t writing to the container filesystem overlay: - Results persistence: disable by default in Prefect 3.x; if you enabled it, back it with a PVC or object storage, not the image layer. - Prefect logs go to API by default; avoid file handlers that write locally. 5) Consider a sizeLimit if you use tmpfs for /tmp to force timely reclaim: - emptyDir:

emptyDir:
       medium: Memory
       sizeLimit: 2Gi

This prevents node-level OOM due to tmpfs growth and nudges reclaim earlier. What your trend should look like - During high activity, dentry will rise and fall; across hours/days it should hover within a band. - It should not linearly increase with every run. If it does, something is still touching overlayfs or pinning dentries. If it’s still growing, please share: - Output of

mount | grep ' /tmp '

from the worker pod - Whether you’re using a fixed working_dir and if so, what’s in it over time - A

slabtop -o

snapshot after a few hours post-change I can help pinpoint whether overlay is still in the path, TMPDIR isn’t inherited by the flow subprocess, or there’s another directory being churned.