@Marvin Why does database gets locked and flow does not show any progress when executing many tasks (4000+) concurrently?

It seems that you're encountering an issue similar to one reported on GitHub. When executing a large number of tasks concurrently, such as over 1500 tasks, users have reported a

QueuePool

error from the database. This could be related to resource limitations, connection pooling constraints, or database locking mechanisms being overwhelmed by the volume and rapidity of concurrent access attempts. To assist you further, here are a few steps you can consider: 1. Review the details of the issue reported on GitHub to see if it matches your situation: Database QueuePool issue with 1500+ task flows. 2. Check the configuration of your database and ensure it can handle the number of concurrent connections your workflow requires. 3. Consider breaking up the workflow into smaller batches of tasks to mitigate the strain on the database. 4. Monitor database performance metrics to understand at what point the locking occurs and adjust resources accordingly. Without additional details on your specific error message or setup, these suggestions are based on common patterns observed in similar issues. If you'd like more tailored advice, feel free to provide more context or error messages.