Hey @Marvin, Is it possible to run replicas of prefect background services?

✅ thought for 291.0 seconds

Ah thanks

if you're hitting scaling issues with 1 replica it'd be super helpful to know what they are as we proceed on this work!

Thanks! the event persister is crazy backed up

Short answer: yes—with a few caveats. In Prefect 3.x you can run the server’s background services in multiple replicas, but not every service benefits from horizontal scaling the same way. What scales well - Event-driven services (safe to run with multiple replicas): Event Persister, Reactive Triggers. These use Redis consumer groups so messages are distributed across replicas without double-processing. What is idempotent but usually best as a single replica - Loop-style services like Scheduler, Mark Late Runs, Foreman, Cancellation Cleanup, Pause Expirations, TaskRunRecorder. Running multiples won’t break correctness (work is idempotent and guarded by DB constraints), but it’s redundant and less efficient. Most users run one replica of these. Recommended deployment pattern - Run API servers without services:

prefect server start --host 0.0.0.0 --no-services

- Run background services separately (one pod for most setups; add replicas primarily for event-driven services if needed):

prefect server services start

Prereqs for replicas/HA - Use PostgreSQL (not SQLite) and run DB migrations once. - Use Redis for messaging/order/leases; point all replicas to the same Redis and Postgres. - Common env vars:

PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@host:5432/prefect>
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage

If you share a bit more about your setup (Kubernetes vs. Docker, Postgres/Redis availability) and which services you want to scale, I can suggest an exact layout and env file for you.

> Event-driven services (safe to run with multiple replicas): Event Persister, Is this part from Marvin accurate? If so simply splitting out background services into multiple things might be highly worthwhile!

yes, some of them are safe to run horizontally but yep currently the

prefect server services start

command runs them all together so you could very well write your own python entrypoints and start specific services per pod until we either allow doing that via the CLI or audit all the services and make sure they can all be horizontally scaled

If it's just hacking on the entrypoint that's good news

yea you can totally start the services individually by calling the python code like how the cli does in aggregate. we should just make that easier eventually in the CLI. and for sure!

Ah I see.... is this possible even with a helm deploy? We can override the entrypoint for the backgroundServices piece? Or I guess we'd basically remove that and run our own services container outside of helm til chart gets an update based on any work done to split out horizontally scalable services?

the latter sounds like what i’d opt for. if you encounter blockers/weirdness with that please let me know and i can try to unblock you

@Nate FYI, bumping this setting way up (20 => 5000) dropped our CPU a bunch: https://docs.prefect.io/v3/api-ref/settings-ref#batch-size-2 Might make sense to default it higher?

we can explore that! defaults are tricky bc the default experience after install is running an ephemeral server as a subprocess but its possible we can safely bump

True, defaulting for scale is not necessarily the right thing to do. Well in that case perhaps https://docs-3.prefect.io/v3/advanced/self-hosted could mention a couple of settings (I imagine we'll find some more) that need to be turned up for prod use?

that's a great suggestion! will get a PR up for that in a bit or else happy to review one

Oh the docs are in the repo, cool I'll get my feet wet tmrw then 🙂