Thanks for the details — that’s frustrating. Based on what you’re seeing (API intermittently “alive but not processing,” health checks not tripping, schedules continue but “emitted”/ad‑hoc runs stall), there are a few likely culprits and some concrete steps that usually stabilize things on Azure Container Apps (ACA). What’s probably happening - The API process is up enough to return a basic health probe but is not actually ready/functional (e.g., DB connectivity or event pipeline is wedged). In that case, a liveness probe hitting a shallow endpoint won’t restart it. - Your services container (scheduler, events, etc.) is still healthy, so scheduled runs continue to be created — but ad‑hoc/“emitted” runs rely on the events/API path and stop when the API is unhealthy. - In Prefect 3.x production, using Redis-backed messaging is important; without it, event-driven pathways (like emitted or triggered runs) can silently degrade after network blips or container restarts. Recommended production pattern for Prefect 3.x in ACA - Split API and services: - API: multiple replicas - Command:

prefect server start --no-services

- Services: exactly 1 replica - Command:

prefect server services start

- Use Postgres (not SQLite) for the API DB. - Use Redis for messaging/events/concurrency (critical for reliability of emitted/triggered runs). - Probes: use

/ready

for readiness AND consider using

/ready

for liveness as well in ACA to catch dependency issues (DB/broker) that

/health

won’t see. ACA probe examples - API app (ingress targetPort 4200):

"probes": {
  "liveness": {
    "httpGet": { "path": "/ready", "port": 4200 },
    "initialDelaySeconds": 60,
    "periodSeconds": 30,
    "timeoutSeconds": 10,
    "failureThreshold": 3
  },
  "readiness": {
    "httpGet": { "path": "/ready", "port": 4200 },
    "initialDelaySeconds": 10,
    "periodSeconds": 5,
    "timeoutSeconds": 5,
    "failureThreshold": 2
  }
}

- Services app (no ingress needed; single replica):

"probes": {
  "liveness": {
    "httpGet": { "path": "/health", "port": 4200 },
    "initialDelaySeconds": 60,
    "periodSeconds": 30,
    "timeoutSeconds": 10
  },
  "readiness": {
    "httpGet": { "path": "/ready", "port": 4200 },
    "initialDelaySeconds": 10,
    "periodSeconds": 5,
    "timeoutSeconds": 5
  }
}

Tip: ACA cold starts can be slow; if you see flapping, bump the initial delays (e.g., liveness 120s, readiness 30s) and increase timeouts to 10–15s. Core environment/config checklist - API database:

PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@host:5432/prefect>

- Messaging / events (recommend Redis in production):

PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
PREFECT_REDIS_MESSAGING_HOST=<your_redis_host>
PREFECT_REDIS_MESSAGING_PORT=6379

- Run commands - API:

prefect server start --no-services

- Services (single replica):

prefect server services start

- Logging (to get more signal in ACA logs when this happens):

PREFECT_LOGGING_LEVEL=DEBUG
# or start with
prefect server start --log-level DEBUG
prefect server services start --log-level DEBUG

Why this helps your exact symptoms - Using

/ready

for liveness in ACA will restart the API when it loses DB/broker connectivity or its internal readiness flips, which

/health

alone will miss. - Keeping services separate (and single-replica) ensures the scheduler and events keep running, while multiple API replicas give you failover for UI/API and ad‑hoc triggers. - Redis-backed messaging prevents “emitted” runs from silently stalling after transient failures. Additional stability checks for ACA

- Resources: make sure the API container has enough memory; intermittent “no logs, stuck until restart” can be OOM or GC pressure. Try 1 CPU / 1–2 GiB RAM to start; watch for OOMKilled in ACA events. - Scale: - API: minReplicas ≥ 2 - Services: minReplicas = maxReplicas = 1 - Graceful shutdowns: give ACA a generous termination grace period so the API can drain (e.g., 60s). If you can share a bit more, I can tailor this precisely: - Prefect version you’re running (e.g.,

prefect version

) - How you’ve split API vs services today (one container or two?) - Your database and whether you’re using Redis for messaging - ACA probe configuration and resource limits/requests - Any ACA “Reason”/“ExitCode” for the API container when it dies If helpful, I can also provide a minimal az CLI/ARM/Bicep example for: - API app (2 replicas,

/ready

for liveness/readiness) - Services app (1 replica, private, no ingress) - Environment variables for Postgres + Redis configured as above

Sorry, I encountered an error while processing your request. Please try again.

Thanks — since you’ve split API and services and already use Redis, the pattern is right. What you’re describing points to the API being “up” enough for a shallow probe but not actually functional (DB/broker wedge, socket exhaustion, or OOM). When that happens, emitted/ad‑hoc events can’t be ingested and you’ll miss them until a restart. Here’s how to both stop the API from silently wedging and avoid losing events during brief outages. Immediate hardening for ACA - Use /ready for liveness and readiness on the API - /health only checks the process; /ready verifies core dependencies. In ACA, make liveness hit /ready so the container is restarted if DB/broker connectivity is gone. - Start with these and tune if you see flapping:

"probes": {
  "liveness": {
    "httpGet": { "path": "/ready", "port": 4200 },
    "initialDelaySeconds": 60,
    "periodSeconds": 30,
    "timeoutSeconds": 10,
    "failureThreshold": 3
  },
  "readiness": {
    "httpGet": { "path": "/ready", "port": 4200 },
    "initialDelaySeconds": 10,
    "periodSeconds": 5,
    "timeoutSeconds": 5,
    "failureThreshold": 2
  }
}

- Run 2+ API replicas (services stays at 1) - This avoids event ingestion downtime if one replica crashes. In ACA set minReplicas ≥ 2 for API; 1 for services. - Give the API enough resources and check for OOM - Start with 1 CPU / 1–2 GiB RAM and watch ACA events for OOMKilled. - Tune server timeouts to play nicely with ACA/LB

PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=25

- Turn up logs while you diagnose

prefect server start --no-services --log-level DEBUG
prefect server services start --log-level DEBUG

Look for DB connection errors, timeouts, or broker disconnects around the time the API “dies”. Reduce or eliminate event loss You’ve got two good paths — do both if events are mission‑critical: 1) Add client-side resilience (quick win) - Wrap

emit_event

with retries and a disk outbox so you can replay if the API is down longer than the client’s retry window. - Example emitter with outbox + backoff:

# emitter.py
import json, os, time, uuid, pathlib, logging
from prefect.events import emit_event

log = logging.getLogger("event-outbox")
OUTBOX_DIR = pathlib.Path(os.environ.get("PREFECT_EVENT_OUTBOX_DIR", "/var/lib/prefect-event-outbox"))
OUTBOX_DIR.mkdir(parents=True, exist_ok=True)

def emit_with_outbox(event: str, resource: dict, payload: dict | None = None, max_retries: int = 10, base_backoff: float = 0.5):
    for i in range(max_retries):
        try:
            emit_event(event=event, resource=resource, payload=payload)
            return
        except Exception as exc:
            wait = min(base_backoff * (2 ** i), 30)
            log.warning(f"emit_event failed (attempt {i+1}/{max_retries}): {exc} - retrying in {wait}s")
            time.sleep(wait)
    # Persist to disk for replay if all retries fail
    fname = OUTBOX_DIR / f"{int(time.time()*1e6)}_{uuid.uuid4().hex}.json"
    fname.write_text(json.dumps({"event": event, "resource": resource, "payload": payload}))
    log.error(f"emit_event exhausted retries; wrote to outbox: {fname}")

- Minimal replayer you can run as a sidecar or cron:

# replayer.py
import json, pathlib, time, logging
from prefect.events import emit_event

log = logging.getLogger("event-replayer")
OUTBOX_DIR = pathlib.Path("/var/lib/prefect-event-outbox")

def replay_once():
    for f in sorted(OUTBOX_DIR.glob("*.json")):
        try:
            data = json.loads(f.read_text())
            emit_event(**data)
            f.unlink()
            <http://log.info|log.info>(f"Replayed and removed {f}")
        except Exception as exc:
            log.warning(f"Replay failed for {f}: {exc}")
            # Stop here to avoid tight loop while API is down
            break

if __name__ == "__main__":
    OUTBOX_DIR.mkdir(parents=True, exist_ok=True)
    while True:
        replay_once()
        time.sleep(5)

- If you emit a high volume of events, also consider batching at the application level and/or increasing client buffering. If you want help tuning those, share your approximate events/sec. 2) Decouple via a durable bus (best for guarantees) - Send your application events to a durable system first (Azure Event Hubs, Kafka, or even Redis streams), then run a small stateless relay (Container App or Function) that reads from the bus and forwards to Prefect. If the Prefect API is down, the relay just retries later and no events are lost. - Advantages: durable storage, replay capability, and simple scaling; your app isn’t coupled to Prefect’s API availability. Server-side event service tuning - If you see backpressure on the Prefect side during spikes, allow a larger in‑memory queue and more flush retries:

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE=100000
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_MAX_FLUSH_RETRIES=10
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=50
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=2

Quick diagnosis checklist - Prefect version (please share

prefect version

): upgrade to the latest 3.x if you’re behind. - ACA probes: confirm both liveness and readiness use

/ready

for the API. - API resources: any OOMKilled or probe flapping in ACA events? - DB connectivity: any asyncpg/DB timeouts in the API logs around failures? - Event volume: rough events/sec to size buffers and relays. - API replicas: confirm minReplicas ≥ 2 for the API app; services = 1. If you can share: - Your API and services Container App YAML/ARM/Bicep snippets (probes + resources) - Prefect version and any API log excerpts around a failure window …I can pinpoint settings and provide a ready‑to‑paste ACA configuration (including a tiny Event Hubs→Prefect relay) tailored to your setup.