Hi guys. I was not able to find any solution for my current problem. Is it possible to delete flow run history from specific day like yesterday or in highly repeated flows delete history older than 5 hours? I ran into problem with my memory on EC2, because my flow generated 11000 flow runs and EC2 instance run out of memory. I need to make some script which removes older runs based on flow id or name. Thanks for your help and ideas. 🙂

something like these:

docker exec -it tmp_postgres_1 sh -c "psql -U prefect -d prefect_server -c \"DELETE FROM task_run WHERE updated < NOW() - INTERVAL '30 days';\" "
docker exec -it tmp_postgres_1 sh -c "psql -U prefect -d prefect_server -c \"DELETE FROM log WHERE updated < NOW() - INTERVAL '30 days';\" "
docker exec -it tmp_postgres_1 sh -c "psql -U prefect -d prefect_server -c \"DELETE FROM task_run_state WHERE updated < NOW() - INTERVAL '30 days';\" "

You might want to look into which tables are taking up the most space and modify the table names and WHERE clauses accordingly. Since your tables have already expanded in size, you'll also want to run a VACUUM FULL after these.

@Anurag Bajpai thank you, so it means that I need to run it manually via docker exec right? I was looking for some script in python which can be run via sh scripts or to schedule it as standalone Prefect task.

@Martin Durkac you can do it from Python as well. You can use PostgresExecute task for it and run it as part of a “maintenance flow” on schedule. But when you wrote your instance is running out of memory, do you mean memory or disk space? If you mean disk space, then deleting some data from the instance may help, but if you mean memory, you may need another solution (scaling up your instance to get one with more memory, or scaling out to a compute cluster like EKS or ECS).

@Anna Geller thank you I will try this. My instance run out of disk space. I had only 20GB of disk space and after 2 weeks my instance stopped working and I had to increase it. I run one EC2 instance with more docker containers with their own agents. Main prefect server runs on EC2 linux environment.

@Martin Durkac an alternative would be to attach another EBS volume to the instance, if disk space is the only constraint. But it requires some work to ensure this attached volume will actually get used, AWS doesn’t do any magic with respect to that 🙂

Hey @Martin Durkac, there’s two things here. First is the database memory, which was discussed here. Second is the

Results

. A lot of people find that

Results

generated by Prefect end up taking a lot of space. You can delete some of the task results in the

.prefect

folder as these can take up a bit of memory.

Hi @Kevin Kho I tried to delete Results manually and without any change in my disk space. I need to try methods mentioned above to delete records from flow history and then I will see if it helps. Nevertheless thank you Kevin it might be helpful in the future 🙂

Without any change in disk space!? I guess you must always return small objects in your flows