@Marvin I am hosting my own prefect server and has set up a postgres for all the flowruns. How come the older flow run records are being removed?

It appears the specific details about flow run record removal in a self-hosted Prefect setup using Postgres are not readily found in the documentation excerpts I retrieved. Prefect Cloud's documentation mentions a "Flow Run Retention Policy" which is a setting pertaining to how long flow runs are retained based on your account tier, but this does not directly apply to a self-hosted Prefect server. I recommend checking the following potential causes for older flow run records being removed: 1. There may be Prefect Server settings or garbage collection processes that automatically clean up old flow run records to prevent unnecessary storage use. These settings could be based on either the number of records or the age of the records. 2. Custom database maintenance tasks or scripts might be running that prune older records from your Postgres database. To understand and configure the data retention policy or identify any automated maintenance tasks, you may want to look further into documentation or community discussions related to data retention and garbage collection in Prefect Server. If you are experiencing unexpected data removal and cannot find the information through documentation, it might be worthwhile to search through the issues on Prefect's GitHub repository or reach out to the community for insight into why this could be occurring. Shall I search GitHub for related issues or do you have a specific area where you would like to further investigate?

@Marvin yes please

I found a couple of GitHub issues that might be relevant to your question about flow run records being removed in Prefect with Postgres: 1. Consider adding DB cleanup scripts to save space in Postgres: This issue discusses the idea of retention rate policies and DB cleanup scripts for managing database size and performance in Prefect 2.0 with Postgres. It appears that there aren't any built-in retention policies, but you can use tools like

pg_cron

for periodic DBA tasks, such as purging old logs. However, caution is advised to avoid breaking references in other tables. - GitHub Issue #5813 2. Error in middleware/server when deleting a flow with >50k flow runs: The user encountered errors when trying to delete a flow with many flow runs. While this isn't directly about automatic removal, it presents issues related to managing large numbers of flow runs within Prefect's infrastructure. - GitHub Issue #11060 These issues might give you more context on how flow run records are managed and potential problems with high volumes of flow runs with Prefect's server and Postgres setup. There doesn't seem to be a default automated retention policy, but custom solutions like database cleanup scripts can be implemented with care.