Javier Ruere

11/01/2022, 5:47 PM
I'm trying to use Prefect 2.6.5 to run a Flow locally. The Flow has >10 Tasks and quite a bit of concurrency (>1000). It's an async Flow and all Tasks are async although several use
internally. I'm using the ConcurrentTaskRunner. The flow is ran like this:
        backend_options={"debug": True, "use_uvloop": True},
The problem I' having is that some Tasks finish but they remain in state Running forever. There's no output I could see to understand the problem. Help, please.
Something which might be relevant is that we are using caching a lot and using PostgreSQL as Orion's DB.

Kalise Richmond

11/01/2022, 10:07 PM
Hi Javier, our team is actively investing in improving the engine reliability in regards to both asynchronous and synchronous task runs in particular to large volume runs. We'll make sure to add this to the list of issues we have seen.

Javier Ruere

11/02/2022, 8:06 AM
Another issue which made me lose a lot of time was that when mixing submitted sync and async tasks, a thread would use 100% CPU making all running tasks much slower.
@Kalise Richmond should I use sync tasks instead? Are they more reliable?


11/03/2022, 8:08 PM
Async tasks should generally be more reliable than sync tasks. Can you share a MRE?

Javier Ruere

11/05/2022, 7:49 PM
Since I posted, I've converted several quick tasks to regular async functions and removed several map() calls, which we used extensively. That appears to side step the problem. It would be hard to produce an MRE as the issue is hard to reproduce consistently; although once it happens, it continues to happen every time.