Thanks for the details. Short answer: there isn’t a built-in

--safe-only

or

--singleton-only

switch. In Prefect 3.x you control which services run by enabling/disabling them with environment variables, then run

prefect server services start

(which will only start the enabled ones). Why you’re seeing the UniqueViolationError - That constraint is hit when two Task Run Recorder consumers try to persist the same task run at the same time. - With multiple “background services” processes running across data centers, it’s possible for Task Run Recorder to race and attempt an insert/upsert for the same

(flow_run_id, task_key, dynamic_key)

. Known reports - #19864 duplicate key violates uq_task_run__flow_run_id_task_key_dynamic_key - #20144 constraint violation with multiple background workers - #20529 IntegrityError on uq_task_run__... on 3.6.13 How to separate services (recommended pattern on 3.6.20) - Run singleton services in one place (one cluster/process). - Only run horizontally safe consumers in other places. - Specifically for 3.6.20, I recommend running exactly one Task Run Recorder instance. You can see what services are available and whether they’re enabled with:

prefect server services ls

Environment variables to control each service (set to true/false): - Scheduler:

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED

- Late runs:

PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED

- Pause expirations:

PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED

- Foreman:

PREFECT_SERVER_SERVICES_FOREMAN_ENABLED

- Repossessor:

PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED

- Cancellation cleanup:

PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED

- DB vacuum:

PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED

(accepts true/false or a set like “events”, “flow_runs”) - Task run recorder:

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED

- Triggers/actions:

PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED

- Event persister:

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED

- Event logger (debug):

PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED

(usually leave false) Which ones should be singleton vs safe to scale - Singleton (run in one place): scheduler, late runs, pause expirations, foreman, repossessor, cancellation cleanup, DB vacuum, and—on 3.6.20—Task Run Recorder (to avoid the unique constraint race). - Horizontally safe: event persister, triggers. Event logger is debug-only. Example: dedicate services to one “primary” cluster Primary (run all singletons and the recorder):

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=true
PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED=true
PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED=true
PREFECT_SERVER_SERVICES_FOREMAN_ENABLED=true
PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=true
PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=true
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=true
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true

prefect server services start

Secondary clusters (disable all singletons and the recorder):

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=false
PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED=false
PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED=false
PREFECT_SERVER_SERVICES_FOREMAN_ENABLED=false
PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=false
PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=false
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=false
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=false

# Optionally keep these enabled if you want more consumers
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true

prefect server services start

Operational tip - If you want an API/UI-only pod/process, you can run the webserver without services: ```

prefect server start --no-services ``` …and then run a separate deployment just for services (with the env vars above to choose which). Questions to help tailor the setup - Are both clusters pointing to the same Postgres database? - Do you want me to share a concrete Helm/Docker Compose snippet showing this split for 3.6.20? If you’d like, I can put together a minimal example values file for a “primary services” release and a “secondary services” release.

Got it — since both clusters point at the same Postgres, you’ll want to run exactly one instance of the singleton services and, to avoid your UniqueViolationError, a single Task Run Recorder. There isn’t a

--safe-only

/

--singleton-only

flag. Instead, enable/disable services via env vars and then run

prefect server services start

. You can confirm the active configuration with:

prefect server services ls

Recommended split for 3.6.20 - Run all singletons and the Task Run Recorder in one “primary” cluster/process. - Disable all singletons and the Task Run Recorder in the “secondary” cluster(s). - It’s fine to run Event Persister and Triggers in multiple places. Primary cluster (services process)

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=true
PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED=true
PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED=true
PREFECT_SERVER_SERVICES_FOREMAN_ENABLED=true
PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=true
PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=true
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=true
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true

prefect server services start

Secondary cluster(s) (services process)

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=false
PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED=false
PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED=false
PREFECT_SERVER_SERVICES_FOREMAN_ENABLED=false
PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=false
PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=false
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=false
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=false

# Optional: safe to leave these on if you want additional consumers
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true

prefect server services start

If you want API/UI-only pods, run:

prefect server start --no-services

Why this helps your error - The UniqueViolationError on

uq_task_run__flow_run_id_task_key_dynamic_key

is caused by multiple Task Run Recorder instances trying to insert the same task run at the same time. Keeping a single recorder instance avoids that race. - Related reports: #19864, #20144, #20529 Notes - Singleton services (scheduler, late runs, pause expirations, foreman, repossessor, cancellation cleanup, DB vacuum) should run in one place. While they use DB coordination to avoid concurrent execution, running them everywhere causes unnecessary contention/noise. - Event Logger (

PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED

) is debug-only; typically leave it false. Want me to share a concrete Helm values or Docker Compose example that implements this split?

Awesome — here are two concrete Docker Compose snippets for Prefect 3.6.20 showing a primary/secondary split. Both clusters point to the same Postgres. The primary cluster runs all singleton services and a single Task Run Recorder; secondary clusters disable those. Primary cluster (runs singletons + Task Run Recorder)

version: "3.9"

services:
  prefect-server-web:
    image: prefecthq/prefect:3.6.20
    command: >
      prefect server start --no-services --host 0.0.0.0 --port 4200
    ports:
      - "4200:4200"
    environment:
      # Shared Postgres across clusters
      - PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@db-host:5432/prefect>
      - PREFECT_SERVER_LOGGING_LEVEL=INFO

  prefect-server-services:
    image: prefecthq/prefect:3.6.20
    command: prefect server services start
    restart: unless-stopped
    environment:
      - PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@db-host:5432/prefect>

      # Singletons ON
      - PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=true
      - PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED=true
      - PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED=true
      - PREFECT_SERVER_SERVICES_FOREMAN_ENABLED=true
      - PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=true
      - PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=true
      - PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=true

      # Task Run Recorder ON (only here to avoid duplicate key races)
      - PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true

      # Safe to scale
      - PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
      - PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true
    depends_on:
      - prefect-server-web

Secondary cluster(s) (disables singletons + Task Run Recorder; keeps safe consumers)

version: "3.9"

services:
  prefect-server-web:
    image: prefecthq/prefect:3.6.20
    command: >
      prefect server start --no-services --host 0.0.0.0 --port 4200
    ports:
      - "4200:4200"
    environment:
      - PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@db-host:5432/prefect>
      - PREFECT_SERVER_LOGGING_LEVEL=INFO

  prefect-server-services:
    image: prefecthq/prefect:3.6.20
    command: prefect server services start
    restart: unless-stopped
    environment:
      - PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@db-host:5432/prefect>

      # Singletons OFF
      - PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=false
      - PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED=false
      - PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED=false
      - PREFECT_SERVER_SERVICES_FOREMAN_ENABLED=false
      - PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=false
      - PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=false
      - PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=false

      # Task Run Recorder OFF (prevents UniqueViolationError across clusters)
      - PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=false

      # Safe to run in multiple places
      - PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
      - PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true
    depends_on:
      - prefect-server-web

Notes - Keep exactly one “primary”

prefect-server-services

instance across all clusters (e.g., scale 1 replica). This avoids the

asyncpg.exceptions.UniqueViolationError

on

uq_task_run__flow_run_id_task_key_dynamic_key

. - Running multiple API/UI instances (

prefect server start --no-services

) is fine; they’re stateless and share the same DB. - In any workers/agents you run, point

PREFECT_API_URL

to a single, load-balanced address for the API, e.g.

http://<lb-or-dns>:4200/api

. - You can verify what each container will run with:

prefect server services ls

If you want, I can adapt these to your existing Compose layout (different network names, secrets, or an external Postgres service block).