Hi there 👋
I’m working on a flow which (in theory) spawns off a lot of tasks in flow runs (about 10-50k task runs each day). I’m sure Prefect can handle such volumes, however there is also a bit of overhead associated with this (database / disk):
• log records for “Created task run”, “Submitted task run” “Execution … immediately”
• task_run record
• task_run_state record
• flow_run record
Right now I got about 380k task runs and 17k flow runs in my database, with already 20GB storage. Understandably this originates by passing data to each sub flow.
I suppose my current setup is not ideal (well unless I want to pay a lot for storage). Regardless of that, is there a way such that Prefect (Orion), performs roll ups on these tables? Or should I just delete these records myself from the database.
✅ 1
Joël Luijmes
12/05/2022, 2:06 PM
I just truncated (with cascaded) the following tables
task_run
flow_run
logs
Prefect still seems to function 🙂 So I suppose just deleting the data works to clean up🤞
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.