https://prefect.io logo
#prefect-community
Title
# prefect-community
y

Young Ho Shin

09/08/2022, 4:28 AM
Hello all. I am running into various
sqlalchemy
errors when running a test flow with many tasks (>10000) locally. Here's the code I'm running: https://gist.github.com/yhshin11/1832bc945446a62c5c6152abb9c1a0a5 It seems like the problem has to do with the fact that there are too many tasks that are trying to write to the Orion database at the same time. I tried switching to a Postgres database as described in the [docs](https://docs.prefect.io/concepts/database/), and also adding concurrency limit of 10. Neither seems to fix the issues. Any ideas about how to fix this? Here's an example of the kind of errors I'm getting:
Copy code
sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: <https://sqlalche.me/e/14/3o7r>)
1
h

Hedgar

09/08/2022, 11:10 AM
Have you tried indicating retries that's delayed by some seconds as arguments to the task decorator of the function? Database connections most times goes dead for inexplicable reasons. But with the possibility of retries your flow run won't stop except it reaches the maximum retries indicated
🙏 1
a

Anna Geller

09/10/2022, 12:15 AM
another alternative to consider is Prefect Cloud where you don't need to worry about this -- I'm aware it may not be up to you to decide if you can use Cloud but just sth to consider
28 Views