Short answer: yes, that Redis Lua panic can happen with Prefect—if you’re using the Redis-backed features and they end up operating on very large sets. Let’s quickly triage and isolate whether Prefect is the source, then we can tune or mitigate. Why this can happen - Prefect 3.x only hits Redis if you’ve enabled any of: - Redis-backed concurrency lease storage - Redis-backed events messaging/causal ordering - Docket backed by Redis - Those components use small Lua scripts for atomic operations. Under extreme cardinalities (e.g., very large sets or many keys in a single eval), Redis’s embedded Lua can raise “PANIC: unprotected error in call to Lua API (table overflow)”. Most likely triggers in Prefect - Events causal ordering (Redis): a script reads followers of an event with

SMEMBERS

and returns them in a Lua table. If a single “leader” accumulates thousands of followers, the script can blow up. - Concurrency lease storage (Redis): a create/revoke script loops over a variable number of keys and does `HSET`s/index cleanup. If a single lease references a very large number of limits, the Lua table can grow too large. First, confirm whether Prefect is actually using Redis Run this to see effective settings:

prefect config view | grep -i 'redis\|events\|concurrency\|docket'

If you see any of these pointing at Redis, Prefect is using it: -

PREFECT_SERVER_EVENTS_MESSAGING_BROKER

/

PREFECT_SERVER_EVENTS_MESSAGING_CACHE

/

PREFECT_SERVER_EVENTS_CAUSAL_ORDERING

-

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE

-

PREFECT_SERVER_DOCKET_URL

Quick isolation steps - If events are on Redis, switch them to memory temporarily and restart the server: - Unset or set these back to memory implementations: -

PREFECT_SERVER_EVENTS_MESSAGING_BROKER=prefect.server.utilities.messaging.memory

-

PREFECT_SERVER_EVENTS_MESSAGING_CACHE=prefect.server.utilities.messaging.memory

-

PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect.server.events.ordering.memory

- If concurrency lease storage is on Redis, switch back to memory to test: -

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect.server.concurrency.lease_storage.memory

- If Docket points at Redis, set: -

PREFECT_SERVER_DOCKET_URL=memory://

If the panic disappears after any of the above, we’ve found the culprit. What to check in Redis - See which keys are growing large:

redis-cli --scan --pattern 'prefect*followers*' | head
redis-cli --scan --pattern 'prefect*concurrency*' | head
# For any suspicious key:
redis-cli MEMORY USAGE <key>
redis-cli SCARD <key>     # for sets
redis-cli ZCARD <key>     # for sorted sets

- Capture the failing script/key names:

redis-cli monitor | grep -i eval

That will show the EVAL/EVALSHA calls and keys being touched around the time of the panic. Mitigations if events/ordering is the source - Ensure you’re not accidentally fanning thousands of “followers” onto a single event. This can happen if many flows subscribe to the same trigger pattern without deduplication. - Trim/rotate old event state; the Redis messaging/ordering layer already trims, but if you find a single “followers” key with huge cardinality, you can safely delete it while the server is stopped to recover:

# Only after stopping Prefect server
redis-cli DEL <the-large-followers-key>

- If you don’t need cross-process causal ordering, keep events on memory to avoid Redis entirely. Mitigations if concurrency leases are the source - Reduce the number of concurrency limits a single run acquires at once; avoid patterns where a lease covers a large number of “resource_ids”. - If you accidentally created many limits/holders, you can clear stale lease state (while the server is stopped) by deleting obviously stale keys under your concurrency prefix. General hardening - Upgrade to the latest Prefect 3.x and prefect-redis packages (bug fixes land there first).

- Use a modern Redis (6.2+/7.0+), ensure sufficient memory, and set sane limits/evictions for your use case. - Watch Redis logs and metrics (latency, memory, commandstats) and consider: -

lua-time-limit

(default is usually fine; your issue is overflow, not timeout) -

maxmemory

and an eviction policy appropriate for your workload - If you’re also hitting Postgres timeouts, that’s usually separate (pool exhaustion or long-running queries). If you can share the exact Postgres error and timing, I can recommend DB-side tuning next. Can you share: - Which of the Redis-backed features above you’ve enabled - Redis version and memory settings - Rough scale (events per second, number of deployments subscribing to events, number of concurrency limits) - A short snippet of the Redis MONITOR output around the panic (redacting secrets) References - Prefect Redis integration (source): prefect-redis repo - Lease storage implementation: lease_storage.py - Events causal ordering: ordering.py - Messaging: messaging.py - Prefect settings overview: Settings docs - Redis Lua scripting background: Redis Lua scripting

- Reduce fan-in (avoid thousands of flows waiting on a single event). - If a single followers set exploded, stop Prefect, delete that specific key to recover, then restart. - Concurrency: - Avoid creating leases that enumerate a very large number of resource IDs/limits. - Clean up obviously stale lease/index keys while Prefect is stopped if they ballooned. - General Redis hygiene: - Upgrade to Redis 6.2+/7.0+, allocate enough memory, set an appropriate

maxmemory

and eviction policy. - Keep

lua-time-limit

default unless you see timeouts (your error is overflow, not time). On your Postgres timeouts Since you also mentioned DB timeouts, double-check pool sizing vs. your server concurrency. Relevant settings: - Pool sizing/timeouts: server/database settings -

PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE

(default 5) -

PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW

(default 10) -

PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT

(default 30s) -

PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE

(default 3600s) -

PREFECT_SERVER_DATABASE_TIMEOUT

(statement timeout, default ~10s) - Engine creation ref: engine configuration - If you run multiple API workers, scale pool_size accordingly or front with PgBouncer (and set

statement_cache_size=0

in

connect_args

for transaction pooling). Can you share: - Which Prefect Redis settings (if any) are enabled in your env - Redis version and whether other services share that Redis - A snippet of Redis SLOWLOG or brief MONITOR around the panic - An example Postgres timeout message (from server logs) I’ll use that to pinpoint whether the crash is from Prefect’s Redis paths and recommend precise fixes.

it would not be a prefect automation, it would be some docket perpetual task like the other background services

EventPersister

etc that is configured via env vars, gets a db connection directly and starts via

prefect server services start

also as a side question, it would be nice to check all prefect related service health in one place (DB, actual server, redis, worker health) in a dashboard, is that something that could be done easily?

also as a side question, it would be nice to check all prefect related service health in one place (DB, actual server, redis, worker health) in a dashboard, is that something that could be done easily?

another interesting point, today no. but you could imagine heartbeat-style events emitted from background services, enabling a "background service health" dashboard in the new oss ui

as someone who's trying to make similar to marvin for my job I feel that