@Marvin what can you tell me about this prefect error: 171938.496 | ERROR | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 60 pending items.

I couldn't find specific documentation or GitHub issues directly related to the error message you provided. However, here are some general resources that might help you troubleshoot Prefect errors involving concurrency and service failures: 1. Troubleshooting Guide: This contains various troubleshooting steps and common issues. Link to troubleshooting guide. 2. Global Concurrency Limits: This guide discusses how to manage concurrency, which might be related to your issue if there's a bottleneck or resource contention. Link to global concurrency limits guide. For more specific investigation, checking the detailed logs and context around the error might provide more clues. If the issue persists, consider creating a GitHub issue with detailed information.

hi @KG - this basically means the client side events worker responsible for publishing events (like logs) is not working for some reason do you happen to have any more logs about this?

Turns out the pod the flow was running on ran out of memory. Log traces don't have much to do with on first glance with EventsWorker service. other than ~60 tasks that were submitted being executed

any chance you could dump that whole stack trace for me? feel free to DM if you want the events worker is responsible for all events, so all state changes like

PENDING

->

RUNNING

for your flow run / task runs so if its foobar'd for some reason, its probably failing to process those initial state changes

First time I'm seeing it. its a new flow .. haven't really used subflows until now. I'll DM you the stacktrace. No worries if you don't get a chance to look at it rn Friday & Sunny in CHI 😎