https://prefect.io logo
#prefect-community
Title
# prefect-community
m

Mihai H

11/28/2022, 9:11 PM
Just quick question... is it possible to control SQLAlchemy/PostgreSQL connections pool size from prefect?
I have a workflow that spins up a lot of tasks (>10k)
I am using PostgreSQL as a storage for prefect
z

Zanie

11/28/2022, 9:19 PM
I’m not sure we expose this. cc @Jake Kaplan
I don’t even see where we are creating a connection pool, so perhaps we’re just using the defaults.
m

Mihai H

11/28/2022, 9:21 PM
Just to be more explicit about the issue I am facing...
z

Zanie

11/28/2022, 9:22 PM
Oh are you not running a standalone Prefect API?
You should run the API separately then each task will contact the API instead of the database (and you’ll use way less connections).
m

Mihai H

11/28/2022, 9:23 PM
No, this is an on prem test app for which I am using Dask as an executor
Any idea regarding a workaround?
z

Zanie

11/28/2022, 9:26 PM
Yeah, start an API with
prefect orion start
then set
PREFECT_API_URL
for your flow run.
@terrence / @Anna Geller do we have official documentation on this pattern?
gratitude thank you 1
👀 1
m

Mihai H

11/28/2022, 9:31 PM
Any hints where I should take a look? All the tutorial I was able to find so far refer on how to use prefect directly
z

Zanie

11/28/2022, 9:37 PM
m

Mihai H

11/28/2022, 9:38 PM
Thanks, that is much appreciated
Bummer, got into exactly same issue
How are you guys mitigating this issue?
Would it be different if I would deploy the app on the cloud?
z

Zanie

11/28/2022, 9:52 PM
If you ran into the same exact issue then it sounds like it was not configured to use the API you started.
Did you set the API URL for your flow run?
j

Jake Kaplan

11/28/2022, 9:55 PM
@Zanie you're right the pool setting isn't exposed. The pool settings would be args to
create_async_engine
(they are used in cloud fwiw)
t

terrence

11/28/2022, 10:25 PM
Some documentation for configuring the database for a standalone Prefect Orion API server here: https://docs.prefect.io/concepts/database/
gratitude thank you 1
I’d need a little more detail on the use case to determine what documentation might be missing. Expanded Dask documentation and examples are on the roadmap.
🙌 1
a

Anna Geller

11/28/2022, 10:31 PM
no docs on that yet, but we definitely need more docs on this kind of setup for OSS/self-hosting. We won't get to it this year but definitely on the roadmap for 2023
m

Mihai H

11/28/2022, 10:32 PM
so unlucky for you :))
because I will have to pester you with all that sort of questions
a

Anna Geller

11/28/2022, 10:34 PM
I have to admit I don't understand what pattern we try to cover here @Zanie
do you mean:
Copy code
prefect orion start --host custom_host_name
then:
Copy code
prefect config set PREFECT_API_URL=custom_host_name
?
r

Ratul Minhaz

11/29/2022, 1:56 AM
@Mihai H did the issue solve for you after setting
PREFECT_API_URL
? I had to set
PREFECT_ORION_DATABASE_TIMEOUT
as well, but for me it was a local setup, with a PostgreSQL database.
m

Mihai H

11/29/2022, 8:09 AM
With those two being set up it works OK so far. I would really like to thank you for your prompt support
🙌 4
5 Views