Young Ho Shin
09/08/2022, 4:28 AMsqlalchemy
errors when running a test flow with many tasks (>10000) locally. Here's the code I'm running:
https://gist.github.com/yhshin11/1832bc945446a62c5c6152abb9c1a0a5
It seems like the problem has to do with the fact that there are too many tasks that are trying to write to the Orion database at the same time. I tried switching to a Postgres database as described in the [docs](https://docs.prefect.io/concepts/database/), and also adding concurrency limit of 10. Neither seems to fix the issues. Any ideas about how to fix this?
Here's an example of the kind of errors I'm getting:
sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: <https://sqlalche.me/e/14/3o7r>)
Hedgar
09/08/2022, 11:10 AMAnna Geller