Hello community. I have a question about two tables in the prefect database. The first table is the
flow_run_state
and the second is the
task_run state
table:
I have ran approximately 40,000 subflows that created 830 million rows in each of these tables (1.66 billion rows in total). Question I have to ask is that this number seems ridiculously high for the amount of subflows that were created. Obviously this leads to a scaling issue where I am filling up database storage on things that are not really that useful. Any thought on this?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.