Sorry, I encountered an error while processing your request. Please try again.

@Marvin we are using prefect 3 v3.6.24 (4 worker server, background services and workpools). We were getting below error in the background services logs.:

Apr 22 10:01:18 <http://ts-mum1-dsr135.alpha-grep.com|ts-mum1-dsr135.alpha-grep.com> bash[129106]: sqlalchemy.exc.IntegrityError: (sqlalchemy.dialects.postgresql.asyncpg.IntegrityError) <class 'asyncpg.exceptions.ForeignKeyViolationError'>: insert or update on table "task_run" violates foreign key constraint "fk_task_run__flow_run_id__flow_run"

❌ Error: GitHub authentication failed

✅ thought for 10.5 seconds

I don’t see the query in this thread. Could you paste it here and share what you’d like me to check (correctness, performance, errors, etc.)? Helpful details: - The full query/command/code in a code block - What you expected vs. what you observed (including any error text) - Your Prefect version (assuming 3.x unless you say otherwise) - How you’re running it (local, deployment, worker type/work pool) - Any relevant logs If it’s long, feel free to attach a file or link to a gist. I’ll review it right away.

Sorry, I encountered an error while processing your request. Please try again.

Thanks for the details. That error means the Task Run Recorder tried to persist a task_run that references a flow_run that no longer exists. In practice, this happens when background services process a task-run event after the corresponding flow run was deleted, or when multiple background-service instances race each other in a multi-replica setup. Most common causes in self-hosted Prefect 3: - Multiple server replicas are running background services at the same time (race conditions on inserts/deletes) - Flow runs are deleted while late/queued task-run events are still being processed by the recorder What to do 1) Ensure only one instance runs background services - Run web servers on all replicas with services disabled:

prefect server start --no-services

- Run background services on a single instance:

prefect server services start

We verified these commands exist with

--help

. 2) Or disable services via env vars on secondary replicas Set these to false on every replica except the one designated to run services:

export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=false
export PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=false
export PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED=false
export PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED=false
export PREFECT_SERVER_SERVICES_FOREMAN_ENABLED=false
export PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=false
export PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=false
export PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=false
export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=false
export PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED=false
export PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=false

3) Verify your effective config

prefect config view --show-defaults | grep PREFECT_SERVER_SERVICES

4) Consider upgrading to the latest 3.6.x Recent releases include improvements to the task-run recorder that help with out-of-order events (e.g., “Remove causal ordering from task run recorder”): - PR #21458 - 3.6.27 release notes Quick check questions - Are you running more than one API server instance with services enabled on each? - Do you have any automated cleanup that deletes flow runs while flows/tasks are still running? If you confirm you have multiple server replicas with services turned on, making the services a singleton (one instance only) typically resolves this specific foreign key violation.

Yep — I pulled a few of the most recent Prefect GitHub items. Highlights: - Issue: Document running multiple background services - Proposes documenting running multiple

prefect server services start

processes for HA/throughput using shared Postgres/Redis and Docket-backed Redis via

PREFECT_SERVER_DOCKET_URL

. - PR: docs: document running multiple background services for HA - Updates the self-hosted scaling docs to cover horizontally scaling background services, coordinating via Docket on Redis, and how to enable/partition services. Closes #21668. - PR: chore(deps): bump uv-dependencies group - Routine dependency updates (click, fastapi, pydantic, uvicorn, ruff, pytest, etc.). Also validated the CLI exists for this area: -

prefect server services

has

ls

,

start

, and

stop

subcommands. Want me to dig into a specific issue/PR or a different repo?