Hello! We recently migrated from Prefect Cloud 1 to Prefect OSS 2. Now we are running Prefect on Fargate. After enabling logging, we started having an issue with one subflow that gets stuck forever, even though it's supposed to finish in seconds. The logs just stop flowing at some point. The timeout doesn't do a thing. Container/db metrics are fine, it's far from even 50% load. Any suggestions? Thanks

Hey ! What’s the taskRunner ? Can this be an network issue ? Like a task hanging out forever ? Logging stopping looks like some issue with the event loop

It's ConcurrentTaskRunner. This doesn't happen with other jobs and it started happening after enabling logging

Hey, theres definitely a lot of variables here. Could explain more what you mean by enabling logging? Do you mean via cloud watch? If you disable logging, does the subflow complete? Do you have logging enabled for other runs?

We enabled logging for the flow runs by using logbook library in our code. Subflow runs fine and completes within seconds if logging is disabled. Logging was enabled for all other flows but we only saw the issue with a single flow.

Is there something unique about that flow? The stuck part might mean something blocking or deadlocking, but difficult to say

The flow is very simple, runs quickly and can easily be run locally. When executed by Prefect it gets stuck all the time and doesn't even react on timeout logic

• Does enabling logging always cause this issue for this flow? • Does disabling logging always let this flow run successfully? • Does this only happen on this particular flow? (there are other flows you can log for) If the answer is yes to all the above, it sounds like there is a particular interaction between this flow code, how it's logging and prefect. Are you able to share a reproduction that will always get stuck?