You don’t need a long‑running poller. In Prefect 3.x you can go fully event-driven so compute only spins up when a new alert arrives. High-level pattern - Emit an event when a new alert row is inserted (from your app/DB trigger). - Use a Prefect Cloud Webhook or direct event emission to Prefect. - An Automation with an EventTrigger listens for that event and runs your deployment. - Run the deployment on a push/serverless work pool so no worker or idle container is needed. Two common ways to wire it up 1) Database/app -> Prefect Cloud Webhook -> Automation -> Deployment (serverless) - Create a webhook (it turns incoming HTTP into Prefect events):

prefect cloud webhook create db-alerts \
    --description "DB alert inserts" \
    --template '{"event": "db.alert.inserted",
                 "resource": {"prefect.resource.id": "alert/{{ body.alert_id }}",
                              "table": "{{ body.table or \"alerts\" }}"},
                 "payload": {{ body | tojson }} }'

This returns a unique HTTPS URL. On every insert, have your DB trigger or the application that writes to the DB POST the new row data to that URL. - Create an Automation with an EventTrigger: - expect: db.alert.inserted - for_each: prefect.resource.id (dedup per alert) - action: Run Deployment (your flow), parameters templated from the event payload (e.g. set

alert_id

to

{{ event.payload.alert_id }}

). UI is easiest for this, or use the API: Create Automation API. Templating reference: Templated parameters. - Run the deployment on a push work pool (serverless) so compute only runs on demand. See Serverless work pools (push). Pick your provider (AWS ECS/Fargate, GCP Cloud Run, Azure Container Instances). Create a pool and optionally provision infra:

prefect work-pool create my-cloud-run-pool --type cloud-run:push
  prefect work-pool provision-infra my-cloud-run-pool

Then deploy your flow to this work pool:

prefect deploy ./flows/alerts.py:process_alert --name process-alerts --work-pool my-cloud-run-pool

2) Application code emits Prefect events directly (no webhook) If your app that writes to the DB is Python and can import Prefect, emit custom events at insert-time:

from prefect.events import emit_event

def on_alert_insert(alert_id: str, metadata: dict):
    emit_event(
        event="db.alert.inserted",
        resource={"prefect.resource.id": f"alert/{alert_id}", "table": "alerts"},
        payload={"alert_id": alert_id, **metadata},
    )

The same Automation (above) will react and run your deployment. Docs: Events and Event triggers. Alternative: create flow runs directly If you prefer to skip events/automations, call the Prefect API to create a flow run when an insert happens:

curl -X POST "$PREFECT_API_URL/flow_runs/" \
  -H "Authorization: Bearer $PREFECT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"deployment_id": "<DEPLOYMENT_ID>", "parameters": {"alert_id": "123"}}'

API reference: Create a flow run. This also pairs well with push work pools to avoid long-lived workers. Why this saves compute - No polling container. Webhook/event triggers fire only on new inserts. - Push/serverless work pools provision ephemeral compute per run, then shut down. Helpful docs - Serverless (push) work pools: Guide - Webhooks: Concepts and Create a webhook

- Events: Events and Create events API - Automations and triggers: Automations, Event triggers, Parameter templating Questions to tailor the setup - Which database are you using (Postgres, MySQL, SQL Server, etc.)? - Can you emit an HTTP request on insert (from a DB trigger or from the app that writes to the DB)? - Are you on Prefect Cloud or self-hosted Prefect Server? - Any preferred runtime (AWS ECS/Fargate, GCP Cloud Run, Azure ACI, Kubernetes)? If you share those, I can give you a step-by-step for your exact stack (including the webhook template and the Automation filter/parameter mapping).

Great question — you can protect yourself from floods so a burst of alerts won’t overwhelm your infra or “kill” a worker. Key controls to add 1) Cap concurrent runs at the work pool (global backpressure) - Prefect will queue extra runs until slots free up. - Set a safe cap for your pool:

prefect work-pool set-concurrency-limit my-pool 5

You can also limit a specific work queue:

prefect work-queue set-concurrency-limit my-queue 5 --pool my-pool

2) Cap concurrency per deployment (local backpressure) - When you deploy the flow, set a per-deployment limit so only N runs of that deployment execute at once:

from prefect.flows import Flow

  Flow.from_source(
      source="<https://github.com/org/repo>",
      entrypoint="flows/alerts.py:process_alert",
  ).deploy(
      name="process-alerts",
      work_pool_name="my-pool",
      concurrency_limit=5,  # per-deployment cap
  )

3) De-duplicate and rate-limit at the trigger - In your Automation’s EventTrigger: - Use

for_each: ['prefect.resource.id']

so only one run per alert ID is launched. - Use a time window (threshold/within) to coalesce bursts (e.g., trigger at most once per 30s) and make your flow pull “all unprocessed alerts” from the DB. That turns a flood into one batched run. - Docs: Event triggers and Templating 4) Prefer push/serverless pools so there’s no long-lived worker to crash - Push pools spin up ephemeral compute per run and respect your concurrency limits. - Guide: Serverless work pools (push) What happens during a flood - Extra events create runs, but your pool/deployment concurrency caps make Prefect queue them; only up to your limit execute concurrently. When slots free up, queued runs proceed. No polling container is needed, and no single worker process is hammered. Optional: batch pattern for floods - Configure the Automation to trigger at most once per X seconds; the flow queries the DB for all “new/unprocessed” alerts and processes them together. This avoids creating one run per alert during spikes. Helpful docs - Serverless pools: Guide - Webhooks: Concepts - Events: Events - Automations: Automations, Event triggers Want me to sketch the exact Automation JSON (with

for_each

and a debounce window) for your setup? If you share: - Cloud vs self-hosted - Push vs pull workers - Your DB (Postgres/MySQL/etc.) I’ll tailor a concrete example.

Got it. Here’s a production-ready approach that keeps your collector as-is, but makes the trigger->run path robust, scalable, and cost-efficient. Recommended production blueprint 1) Decouple DB triggers from HTTP calls - Triggers doing HTTP in production can be brittle and hard to retry/observe. - Prefer having the collector (or a tiny sidecar service) POST to a Prefect Cloud Webhook when it writes the alert row. This keeps your DB purely transactional and gives you clearer retry/backoff logic in app code. - Webhook docs: Webhooks 2) Emit a well-structured event per alert - Create a Cloud webhook that turns HTTP into a Prefect event. Use a stable resource id and include client_id in labels/payload so you can dedupe and batch by client.

prefect cloud webhook create db-alerts \
    --description "DB alert inserts" \
    --template '{"event": "db.alert.inserted",
                 "resource": {"prefect.resource.id": "alert/{{ body.client_id }}/{{ body.alert_id }}",
                              "client_id": "{{ body.client_id }}",
                              "table": "alerts"},
                 "payload": {{ body | tojson }} }'

- Your collector (not the DB trigger) POSTs the new alert to the webhook URL. 3) Use an Automation with an EventTrigger to control bursts - Listen for

db.alert.inserted

. - Set

for_each

to

client_id

so you control concurrency per client. - Use a small time window to coalesce floods into batch runs (e.g., one run per client every 15–60s). The flow then pulls “all unprocessed alerts” for that client in that window. - Docs: Automations, Event triggers, templating: Parameter templating 4) Run on a push/serverless pool so there’s no long-lived worker - Serverless work pools spin up ephemeral compute only when an event triggers a run. - Create and provision a push pool (example for Cloud Run):

prefect work-pool create my-cloud-run-pool --type cloud-run:push
  prefect work-pool provision-infra my-cloud-run-pool

- Deploy your flow to this pool:

prefect deploy ./flows/alerts.py:process_alerts_for_client \
    --name process-alerts \
    --work-pool my-cloud-run-pool

- Serverless guide: Serverless work pools 5) Add backpressure and safety limits - Pool-level cap (global):

prefect work-pool set-concurrency-limit my-cloud-run-pool 10

- Optional queue-level cap:

prefect work-queue set-concurrency-limit my-queue 5 --pool my-cloud-run-pool

- Per-deployment cap (local):

from prefect.flows import Flow

  Flow.from_source(
      source="<https://github.com/org/repo>",
      entrypoint="flows/alerts.py:process_alerts_for_client",
  ).deploy(
      name="process-alerts",
      work_pool_name="my-cloud-run-pool",
      concurrency_limit=5,  # per-deployment cap
  )

This ensures floods queue instead of overwhelming compute, and only up to your caps run concurrently. 6) Make the flow batch- and idempotency-friendly - Parameters:

client_id

, optional

since

watermark. - On start, atomically claim a client-level “batch” via a DB lock or watermark row; then select all rows with status=new (or inserted_after watermark) and process in chunks. - Mark processed rows and update watermark in a transaction; early-exit if nothing to do (so you pay near-zero when quiet). - Idempotency: enforce a unique constraint on an output table keyed by

alert_id

or do “upsert if not exists” per alert. The flow can safely re-run the same alerts without duplication. 7) Per-client single-flight (if required) - The Automation’s

for_each: client_id

+ batching often suffices. - If you need strict “one run per client at a time,” you can either:

- Use a DB lock keyed by client_id around the batch, or - Create separate work queues per client with concurrency=1 (practical for a small, known set of clients). 8) Reliability and security - Webhook hygiene: rotate if exposed:

prefect cloud webhook rotate <WEBHOOK_ID>

- Monitor webhook failures (Cloud emits

prefect-cloud.webhook.failed

). - Prefer app-originated HTTP with retries/backoff over DB triggers to avoid tying transaction success to an external HTTP call. Putting it together - Collector writes to DB and POSTs the alert JSON to the Prefect webhook. - Webhook emits

db.alert.inserted

with

client_id

and stable resource id. - Automation coalesces per client and runs the deployment with

client_id

. - Flow pulls and processes all new alerts for that client, then exits. - Serverless/push pool provisions compute just for that run; concurrency caps ensure graceful backpressure during floods. If you share: - Prefect Cloud vs self-hosted - Your serverless target (Cloud Run, ECS/Fargate, ACI, K8s) - DB type (Postgres/MySQL/etc.) and typical alert rates I can give you a concrete Automation config and a sample flow skeleton (including the DB locking/watermark pattern and batching).

Short answer: a push/serverless work pool is the best fit. It will spin up ephemeral compute per event and can comfortably handle 40–100 concurrent runs as long as your cloud account quotas allow it. Pick the pool type based on your cloud: - GCP: Cloud Run push work pool - Pros: fast cold starts, simple networking, scale-to-zero - Create/provision

prefect work-pool create db-events-pool --type cloud-run:push
    prefect work-pool provision-infra db-events-pool
    prefect work-pool set-concurrency-limit db-events-pool 100

- AWS: ECS (Fargate) push work pool - Pros: strong VPC controls, private subnets/EFS support - Notes: cold starts typically slower than Cloud Run; ensure Fargate/ENI quotas are high enough - Azure: Azure Container Instances push work pool - Pros: managed, scale-to-zero; ensure regional quota for container groups Why push/serverless - No long-lived worker to keep running or to crash under floods - Each flow run becomes a separate serverless job/task; Prefect queues overflow beyond your caps - You control max parallelism via pool and deployment limits What to set for 40–100 parallel runs - Pool concurrency (global cap):

prefect work-pool set-concurrency-limit db-events-pool 100

- Optional queue cap (if you use a specific work queue):

prefect work-queue set-concurrency-limit my-queue 100 --pool db-events-pool

- Per-deployment cap:

from prefect.flows import Flow

  Flow.from_source(
      source="<https://github.com/org/repo>",
      entrypoint="flows/alerts.py:process_alerts_for_client",
  ).deploy(
      name="process-alerts",
      work_pool_name="db-events-pool",
      concurrency_limit=100,
  )

Production notes - Provider quotas: request sufficient limits (Cloud Run jobs/CPU, ECS tasks/ENIs, ACI container groups) to sustain 100 concurrent jobs. - Cold start: Cloud Run tends to be fastest; ECS/ACI are fine but may start slower—if your flows are extremely short-lived, favor faster cold start. - Networking: pick ECS if you need tight VPC/subnet control; Cloud Run is simplest for public-to-private via connectors. - Backpressure: keep your Automation/EventTrigger plus batching/for_each to coalesce floods; Prefect will queue overflow beyond your concurrency limits. Docs - Serverless work pools (push): Guide - Automations / Event triggers: Automations and Event triggers If you tell me your cloud (GCP/AWS/Azure), typical per-run CPU/RAM, and run duration, I’ll give you a concrete pool setup (including job template knobs) to reliably hit 100-way parallelism.