https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> In Prefect 3.7.2 self hosted kubernet...
# ask-marvin
r
@Marvin In Prefect 3.7.2 self hosted kubernetes with redis as a subchart defined based on the prefect-helm chart repo, I see large disk usage. So something like 80GB out of 95G when running df on /bitnami/redis/data. Note that those sizes are well beyond what I specified in the resouces.
m
✅ thought for 177.7 seconds
This is very likely a Redis persistence/storage issue, not Prefect itself storing 80 GB of useful state. A few important distinctions: * 
resources
in Kubernetes control CPU/memory requests/limits. * The 
95G
you’re seeing from 
df
under 
/bitnami/redis/data
is typically the mounted volume size for the Redis PVC. * So it’s normal for 
df
to show the full filesystem backing the PVC, even if you expected smaller runtime resources. For Prefect 3.x, Redis is generally used for coordination/messaging/cache-style data, and that data volume is usually nowhere near tens of GB. The most common reason for large disk usage in Bitnami Redis is: * AOF persistence (
appendonly.aof
) growing over time * sometimes combined with 
dump.rdb
* or temporary rewrite files during AOF compaction What I’d check first Exec into the Redis pod and see which file is actually using space:
Copy code
bash
kubectl exec -it <redis-pod-name> -- bash
du -sh /bitnami/redis/data/*
redis-cli CONFIG GET appendonly
redis-cli INFO persistence
redis-cli INFO memory | grep used_memory
What to look for: * huge 
appendonly.aof
→ AOF is the main culprit * 
dump.rdb
large but reasonable → snapshot persistence * 
temp-rewriteaof-*
files → incomplete/failed AOF rewrite * low Redis memory but huge disk → confirms persistence artifact, not active dataset Why this happens with Bitnami Redis Bitnami Redis commonly enables persistence, and Redis disk usage can exceed the in-memory dataset because: * AOF logs every write * rewrite/compaction can temporarily create extra files * if no 
maxmemory
is set, Redis may keep accepting data until other constraints stop it * stream-heavy workloads can generate lots of write traffic even when retained live data is modest What to do If you do not need durable Redis persistence for Prefect coordination data, the simplest fix is usually to disable AOF and keep Redis lightweight. Example values to review in your Helm config:
Copy code
yaml
redis:
  enabled: true
  master:
    persistence:
      enabled: true
      size: 10Gi
    configuration: |
      appendonly no
      maxmemory 2gb
      maxmemory-policy allkeys-lru
      save "900 1"
Notes: * 
appendonly no
usually cuts disk growth dramatically * 
maxmemory
prevents Redis from growing unbounded in memory * PVC 
size
should be set explicitly if you don’t want a large backing volume If you don’t need persistence at all for this Redis instance, you could also consider disabling Redis persistence entirely, depending on your tolerance for restart data loss. Most useful next step Please paste the output of:
Copy code
bash
du -sh /bitnami/redis/data/*
redis-cli CONFIG GET appendonly
redis-cli INFO persistence
That will tell us pretty quickly whether this is: 1. AOF growth 2. RDB snapshots 3. temp rewrite files 4. something less common I wasn’t able to get useful GitHub issue/discussion results from the tools due to GitHub auth/access problems, so I’d base the diagnosis on the live Redis files first.
r
@Marvin running du -sh /bitnami/redis/data/* gives no such file or directory
m
✅ thought for 46.5 seconds
That usually means one of these is true: * the directory is actually empty, so the shell wildcard 
*
fails * Redis is using a different data dir * Redis 7 AOF files are under an 
appendonlydir
subdirectory * you may be in the wrong container/pod Try these safe checks instead. 1. Ask Redis where its data dir is
Copy code
bash
redis-cli --no-auth-warning CONFIG GET dir
redis-cli --no-auth-warning CONFIG GET appendonly
redis-cli --no-auth-warning CONFIG GET appendfilename
redis-cli --no-auth-warning CONFIG GET dbfilename
That will tell us: * actual working directory * whether AOF is enabled * expected filenames *2. List the directory without using `*`*
Copy code
bash
ls -la /bitnami/redis/data
ls -laA /bitnami/redis/data
du -sh /bitnami/redis/data
find /bitnami/redis/data -maxdepth 3 -type f -exec ls -lh {} \;
If Redis 7 AOF is enabled, also check:
Copy code
bash
ls -laA /bitnami/redis/data/appendonlydir
find /bitnami/redis/data/appendonlydir -type f -exec ls -lh {} \;
3. If Redis says it uses another dir, inspect that one Example:
Copy code
bash
REDIS_DIR=$(redis-cli --no-auth-warning CONFIG GET dir | tail -1)
echo $REDIS_DIR
ls -laA "$REDIS_DIR"
find "$REDIS_DIR" -maxdepth 3 -type f -exec ls -lh {} \;
du -sh "$REDIS_DIR"
4. Confirm you’re in the actual Redis container
Copy code
bash
ps aux | grep redis-server
mount | grep -i redis
df -h
If this is a multi-container pod, 
kubectl exec
may have put you in a sidecar/init container instead of the Redis container. In that case use:
Copy code
bash
kubectl exec -it <pod-name> -c <redis-container-name> -- bash
What I suspect most right now Most likely one of these: * the files are in 
/bitnami/redis/data/appendonlydir
* Redis is configured with a different 
dir
* the data directory is empty but the mounted filesystem itself is large, so 
df
shows 95G capacity even though files aren’t there Please paste the output of these exact commands:
Copy code
bash
redis-cli --no-auth-warning CONFIG GET dir
redis-cli --no-auth-warning CONFIG GET appendonly
redis-cli --no-auth-warning CONFIG GET appendfilename
redis-cli --no-auth-warning CONFIG GET dbfilename
ls -laA /bitnami/redis/data
find /bitnami/redis/data -maxdepth 3 -type f -exec ls -lh {} \;
du -sh /bitnami/redis/data
and I can help interpret what’s actually consuming the space.
r
@Marvin I need to login with the password which I know, but I don't know how to do the login itself
m
✅ thought for 298.8 seconds
r
@Marvin CONFIG GET appendonly returns "yes" CONFIG GET dir gives "/data"
m
✅ thought for 11.0 seconds
That’s very helpful — your Redis data is under /data, not 
/bitnami/redis/data
. And since 
appendonly
is 
"yes"
, the most likely cause is AOF persistence growth. Please run these next:
Copy code
bash
ls -laA /data
find /data -maxdepth 3 -type f -exec ls -lh {} \;
du -sh /data
du -sh /data/*
redis-cli --no-auth-warning INFO persistence
redis-cli --no-auth-warning INFO memory | grep used_memory
A few things to expect: * On Redis 7, AOF may be stored in 
/data/appendonlydir/
* You may see files like: * 
appendonly.aof...
* 
appendonlydir/*.aof.*
* 
dump.rdb
If you want a quick targeted check:
Copy code
bash
ls -laA /data/appendonlydir
du -sh /data/appendonlydir
find /data/appendonlydir -type f -exec ls -lh {} \;
What this means If 
/data
contains very large AOF files, then: * Prefect itself is probably not holding 80 GB of active data * Redis is persisting a long history of write operations * disk usage can be much larger than live memory usage A very telling comparison is: * 
INFO memory
→ active in-memory dataset * size of files under 
/data
→ persisted history/snapshots If memory is modest but 
/data
is huge, that strongly confirms AOF growth. Likely fix For Prefect’s Redis, many setups do not need AOF durability. If that’s true for you, you can usually reduce disk usage by disabling AOF in your Redis chart values. Something like:
Copy code
yaml
redis:
  enabled: true
  master:
    configuration: |
      appendonly no
      maxmemory 2gb
      maxmemory-policy allkeys-lru
      save "900 1"
Depending on your chart version, that may need to be set under a slightly different values path, but the key point is: * disable 
appendonly
* set 
maxmemory
* optionally keep lightweight RDB snapshots only Important caution Don’t delete files manually from 
/data
while Redis is running. That can corrupt persistence state. Better options are: * inspect first * change config * restart Redis cleanly * or trigger a proper AOF rewrite if you decide to keep AOF If you paste the output of:
Copy code
bash
find /data -maxdepth 3 -type f -exec ls -lh {} \;
du -sh /data
redis-cli --no-auth-warning INFO persistence
redis-cli --no-auth-warning INFO memory | grep used_memory
I can tell you exactly whether it’s AOF, RDB, or something else.
If your Redis requires auth, use 
redis-cli
with `-a`:
Copy code
bash
redis-cli -a '<your-password>'
If you’re already inside the Redis pod, that should usually work. Then run:
Copy code
bash
CONFIG GET dir
CONFIG GET appendonly
CONFIG GET appendfilename
CONFIG GET dbfilename
INFO persistence
INFO memory
If you want to do it in one shot from Kubernetes:
Copy code
bash
kubectl exec -it <redis-pod-name> -- redis-cli -a '<your-password>'
If the pod has multiple containers:
Copy code
bash
kubectl exec -it <redis-pod-name> -c <redis-container-name> -- redis-cli -a '<your-password>'
If your password contains special shell characters, wrap it in single quotes as above. If you don’t know the pod name, get it with:
Copy code
bash
kubectl get pods
or in the namespace:
Copy code
bash
kubectl get pods -n <namespace>
If you want, I can also help you find the Redis password from the Kubernetes secret used by the Bitnami Redis chart.
r
@Marvin The memory usage seems reasonable. Let's assume the problem is from appendonly. What would that subchart look like? Presumably the memory lines you have above are not needed. Is the save "900" "1" still needed -- what does it do? And what do I need to do to remove/deleted/purge the existing disk usage from appendonly
m
✅ thought for 20.1 seconds
Yes — if memory usage looks normal and disk is huge, appendonly is the likely culprit. Short answers: * No, the memory settings are not strictly required to fix this specific issue. * No, 
save "900 1"
is not required if you don’t want RDB snapshots. * 
save "900 1"
means: create an RDB snapshot if at least 1 write happened in the last 900 seconds. * To fully reclaim existing AOF disk usage, the cleanest path is usually: 1. disable AOF in the Redis config 2. restart the Redis pod/statefulset 3. if files remain on the PVC, delete them while Redis is stopped or recreate the PVC What 
save "900 1"
does Redis persistence has two main modes: * AOF (
appendonly yes
) logs every write operation * RDB snapshots (
save ...
) periodically writes a point-in-time snapshot This line:
Copy code
yaml
save "900 1"
means: * if at least 1 key changed * within 900 seconds * Redis will write an RDB snapshot So if you want some persistence but much less disk churn than AOF, keeping 
save "900 1"
is reasonable. If you want no persistence at all, then disable both AOF and RDB:
Copy code
yaml
appendonly no
save ""
Recommended subchart config If this Redis is only for Prefect coordination/caching and you’re okay losing Redis state on restart, I’d use:
Copy code
yaml
redis:
  enabled: true
  architecture: standalone

  master:
    persistence:
      enabled: true
      size: 10Gi
    configuration: |
      appendonly no
      save ""
If you do want lightweight snapshot persistence:
Copy code
yaml
redis:
  enabled: true
  architecture: standalone

  master:
    persistence:
      enabled: true
      size: 10Gi
    configuration: |
      appendonly no
      save "900 1"
And yes: if your only goal is to stop the giant appendonly growth, the 
maxmemory
lines are optional. They’re just a good guardrail, not required for this fix. If you want those guardrails too:
Copy code
yaml
redis:
  enabled: true
  architecture: standalone

  master:
    persistence:
      enabled: true
      size: 10Gi
    configuration: |
      appendonly no
      save "900 1"
      maxmemory 2gb
      maxmemory-policy allkeys-lru
How to remove existing appendonly disk usage Important: do not delete AOF files while Redis is running. You have a few options: Option A: Cleanest, if data loss in Redis is acceptable 1. update Helm values to disable AOF 2. scale down/stop Redis 3. remove old AOF files from the volume 4. start Redis again Typical flow:
Copy code
bash
helm upgrade <release> <chart> -f values.yaml
kubectl scale statefulset <redis-statefulset-name> --replicas=0
Then mount/access the PVC from a cleanup pod, or if the volume is retained and mounted elsewhere, remove files like:
Copy code
bash
/data/appendonlydir
/data/appendonly.aof
Then bring Redis back:
Copy code
bash
kubectl scale statefulset <redis-statefulset-name> --replicas=1
Option B: Easiest full reset If Redis data is disposable, delete the Redis PVC and let it be recreated. This is often the simplest way to reclaim space completely. Something like:
Copy code
bash
kubectl delete pvc <redis-pvc-name>
But only after Redis is stopped, and only if you’re comfortable wiping Redis state entirely. Option C: Keep Redis data but stop using AOF Disable AOF and restart Redis. In some cases Redis will no longer use the AOF files, but the old files may still exist on disk until you remove them manually or recreate the volume. So this may stop future growth without reclaiming all old space immediately. What I’d recommend for Prefect For a self-hosted Prefect Redis used as an internal dependency, I’d usually choose one of: * No persistence:
Copy code
yaml
appendonly no
save ""
or * Minimal snapshot persistence:
Copy code
yaml
appendonly no
save "900 1"
If this Redis is not serving as a business-critical datastore, AOF is usually overkill. One caution on Bitnami chart values
Depending on your exact Redis chart version, config may live under 
master.configuration
, or under auth/persistence sections with slightly different names. So if you want, paste your current Redis values block and I can rewrite it precisely. Also, if you want to change this through Helm CLI instead of a values file, I can help with that too. If you can share your current 
values.yaml
Redis section, I’ll give you the exact replacement block.
3 Views
Open in Slack
PreviousNext
Powered by