https://prefect.io logo
Title
j

Joël Luijmes

12/05/2022, 10:44 AM
Hi there 👋 I’m working on a flow which (in theory) spawns off a lot of tasks in flow runs (about 10-50k task runs each day). I’m sure Prefect can handle such volumes, however there is also a bit of overhead associated with this (database / disk): • log records for “Created task run”, “Submitted task run” “Execution … immediately” • task_run record • task_run_state record • flow_run record Right now I got about 380k task runs and 17k flow runs in my database, with already 20GB storage. Understandably this originates by passing data to each sub flow. I suppose my current setup is not ideal (well unless I want to pay a lot for storage). Regardless of that, is there a way such that Prefect (Orion), performs roll ups on these tables? Or should I just delete these records myself from the database.
1
I just truncated (with cascaded) the following tables
task_run
flow_run
logs
Prefect still seems to function 🙂 So I suppose just deleting the data works to clean up🤞
🙌 1