Just quick question is it possible to control SQLAlchemy Pos

Question

Just quick question is it possible to control SQLAlchemy PostgreSQL connections pool size from prefect

Mihai H · Answer

I have a workflow that spins up a lot of tasks gt 10k

Mihai H · Answer

I am using PostgreSQL as a storage for prefect

Zanie · Answer

I m not sure we expose this cc < Jake Kaplan>

Zanie · Answer

I don t even see where we are creating a connection pool so perhaps we re just using the defaults

Mihai H · Answer

Just to be more explicit about the issue I am facing

Zanie · Answer

Oh are you not running a standalone Prefect API

Zanie · Answer

You should run the API separately then each task will contact the API instead of the database and you ll use way less connections

Mihai H · Answer

No this is an on prem test app for which I am using Dask as an executor

Mihai H · Answer

Any idea regarding a workaround

Zanie · Answer

Yeah start an API with `prefect orion start` then set `PREFECT API URL` for your flow run

Zanie · Answer

< terrence> < Anna Geller> do we have official documentation on this pattern

Mihai H · Answer

Any hints where I should take a look All the tutorial I was able to find so far refer on how to use prefect directly

Zanie · Answer

There s some content in the tutorial at <https docs prefect io tutorials deployments run a prefect orion server>

Mihai H · Answer

Thanks that is much appreciated

Mihai H · Answer

Bummer got into exactly same issue

Mihai H · Answer

How are you guys mitigating this issue

Mihai H · Answer

Would it be different if I would deploy the app on the cloud

Zanie · Answer

If you ran into the same exact issue then it sounds like it was not configured to use the API you started

Zanie · Answer

Did you set the API URL for your flow run

Jake Kaplan · Answer

< Zanie> you re right the pool setting isn t exposed The pool settings would be args to `create async engine` they are used in cloud fwiw

terrence · Answer

Some documentation for configuring the database for a standalone Prefect Orion API server here <https docs prefect io concepts database >

terrence · Answer

I d need a little more detail on the use case to determine what documentation might be missing Expanded Dask documentation and examples are on the roadmap

Anna Geller · Answer

no docs on that yet but we definitely need more docs on this kind of setup for OSS self hosting We won t get to it this year but definitely on the roadmap for 2023

Mihai H · Answer

so unlucky for you

Mihai H · Answer

because I will have to pester you with all that sort of questions

Anna Geller · Answer

I have to admit I don t understand what pattern we try to cover here < Zanie>

Anna Geller · Answer

do you mean ```prefect orion start host custom host name``` then ```prefect config set PREFECT API URL=custom host name```

Anna Geller · Answer

< Mihai H> the best resource I have for you now is this

Anna Geller · Answer

Ratul Minhaz · Answer

< Mihai H> did the issue solve for you after setting `PREFECT API URL` I had to set `PREFECT ORION DATABASE TIMEOUT` as well but for me it was a local setup with a PostgreSQL database

Mihai H · Answer

With those two being set up it works OK so far I would really like to thank you for your prompt support