Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi, wrt keeping the database size and performance good, is there a way to set some sort of retention rate policy on the database that maintains the flows, flowruns, etc?

I'm referring to a Postgres database for 2.0

you may use e.g. <https://github.com/citusdata/pg_cron|pg_cron> or something like that to periodically run some DBA tasks such as purging old logs, but be careful about which data you delete, e.g. purging logs should be fine, but purging flow runs you need to be careful to not break other tables that reference that flow run ID

btw I thought you are on Cloud 2.0? if so, you don't have to worry at all about DBA tasks and retention policies

I'm running my own Postgres and gui. There are no on delete cascade foreign keys?

Looks fine to me actually. Some depend rows are deleted and some are set to null. No doubt there are some bugs here and there, but generally structure looks ok. (Referring to bugs as at some point I ran into the issue where I couldn't reset the database due to foreign key constraints. ) :slightly_smiling_face:

yup, FK constraints is exactly what I meant - if not doing it carefully you may cause issues

I guess it somewhere on the roadmap to provide some cleanup scripts to allow proper cleanup of the database?

<@ULVA73B9P> open "Consider adding DB cleanup scripts to save space in Postgres"

<https://github.com/PrefectHQ/prefect/issues/5813>