Hello everyone,
I'm having some issues with the orion.db growing too large.
Context:
I'm using Prefect Orion 2.06b with a persistent local sqlite database with the environment variable:
export PREFECT_ORION_DATABASE_CONNECTION_URL="sqlite+aiosqlite:////home/sqlite_database/orion.db"
The culprit for the growing size seems to be that I use large Pandas DataFrames and pass them through several flows.
Is there any way to reduce the size of the orion.db? Can the output of a nested flow be deleted upon completion of the parent flow?
Thanks for your help!
a
Anna Geller
07/02/2022, 4:58 PM
We are aware of the problem and we have an issue open internally to allow disabling task run results persistence. Until then you may use Postgres DB instead
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.