<@ULVA73B9P> I’m seeing a lot of logs referring to...
# ask-marvin
t
@Marvin I’m seeing a lot of logs referring to missing events while running a flow in kubernetes:
prefect.events.clients - Added event id=019a06fe-62b6-76c4-9bd3-c19505ee4969 to unconfirmed events list. There are now 26 unconfirmed events.
Are those messages somehow harmful?
m
thought for 832.4 seconds
Short answer: those messages are normal and generally not harmful. What they mean - In Prefect 3, events are sent asynchronously to the server over a websocket. Each event is kept in an “unconfirmed” list until the server acknowledges it. - The log you’re seeing just records that an event was queued and is awaiting ack. Under load or with a bit of network latency you’ll see that counter go up and down as batches are sent and acks arrive. When to worry - If the “unconfirmed events” count grows steadily and never comes back down, or you start seeing warnings/errors like failed retries, backfill loops, or “dropping” events on shutdown, then it’s worth investigating connectivity. - In Kubernetes, the most common culprits are: - Egress/network policies blocking or throttling outbound websockets to the Prefect API - Proxies/load balancers with short idle timeouts that terminate websockets - Very short pod termination grace periods that don’t give the client time to flush on shutdown Practical tips - It’s safe to ignore these messages if the count fluctuates and you don’t see errors; they don’t affect flow execution. The worst case if events can’t be delivered is missing some UI timeline entries/automations—not failed runs. - To reduce noise, you can raise the log level for that logger:
Copy code
import logging
  logging.getLogger("prefect.events.clients").setLevel(logging.WARNING)
- For Kubernetes: - Ensure outbound HTTPS/websocket access to the Prefect API is allowed by NetworkPolicy/egress rules. - If you run behind a proxy/LB, set reasonable websocket idle timeouts (e.g., several minutes). - Give pods a sufficient terminationGracePeriodSeconds so the process can flush events on shutdown. Docs for more detail - Events concepts in Prefect 3: Events concepts - API references around events and the client: - Events client API - Create events API - Event ordering If you’re seeing the counter only increase or any error logs, share a snippet and details about where you’re running (Cloud vs self-hosted, any proxies/network policies) and I can help pinpoint the cause.