https://prefect.io logo
Join Slack
Powered by
Just quick question... is it possible to control ...
# ask-community
m
Just quick question... is it possible to control SQLAlchemy/PostgreSQL connections pool size from prefect?
I have a workflow that spins up a lot of tasks (>10k)
I am using PostgreSQL as a storage for prefect
z
I’m not sure we expose this. cc @Jake Kaplan
I don’t even see where we are creating a connection pool, so perhaps we’re just using the defaults.
m
Just to be more explicit about the issue I am facing...
z
Oh are you not running a standalone Prefect API?
You should run the API separately then each task will contact the API instead of the database (and you’ll use way less connections).
m
No, this is an on prem test app for which I am using Dask as an executor
Any idea regarding a workaround?
z
Yeah, start an API with 
prefect orion start
then set 
PREFECT_API_URL
for your flow run.
@terrence / @Anna Geller do we have official documentation on this pattern?
gratitude thank you 1
👀 1
m
Any hints where I should take a look? All the tutorial I was able to find so far refer on how to use prefect directly
z
m
Thanks, that is much appreciated
Bummer, got into exactly same issue
How are you guys mitigating this issue?
Would it be different if I would deploy the app on the cloud?
z
If you ran into the same exact issue then it sounds like it was not configured to use the API you started.
Did you set the API URL for your flow run?
j
@Zanie you're right the pool setting isn't exposed. The pool settings would be args to 
create_async_engine
(they are used in cloud fwiw)
t
Some documentation for configuring the database for a standalone Prefect Orion API server here: https://docs.prefect.io/concepts/database/
gratitude thank you 1
I’d need a little more detail on the use case to determine what documentation might be missing. Expanded Dask documentation and examples are on the roadmap.
🙌 1
a
no docs on that yet, but we definitely need more docs on this kind of setup for OSS/self-hosting. We won't get to it this year but definitely on the roadmap for 2023
m
so unlucky for you :))
because I will have to pester you with all that sort of questions
a
I have to admit I don't understand what pattern we try to cover here @Zanie
do you mean:
Copy code
prefect orion start --host custom_host_name
then:
Copy code
prefect config set PREFECT_API_URL=custom_host_name
?
r
@Mihai H did the issue solve for you after setting 
PREFECT_API_URL
? I had to set 
PREFECT_ORION_DATABASE_TIMEOUT
as well, but for me it was a local setup, with a PostgreSQL database.
m
With those two being set up it works OK so far. I would really like to thank you for your prompt support
🙌 4
7 Views
Open in Slack
PreviousNext
Powered by