Hi there 👋
I’m working on a flow which (in theory) spawns off a lot of tasks in flow runs (about 10-50k task runs each day). I’m sure Prefect can handle such volumes, however there is also a bit of overhead associated with this (database / disk):
• log records for “Created task run”, “Submitted task run” “Execution … immediately”
• task_run record
• task_run_state record
• flow_run record
Right now I got about 380k task runs and 17k flow runs in my database, with already 20GB storage. Understandably this originates by passing data to each sub flow.
I suppose my current setup is not ideal (well unless I want to pay a lot for storage). Regardless of that, is there a way such that Prefect (Orion), performs roll ups on these tables? Or should I just delete these records myself from the database.