Deceivious
08/02/2023, 9:32 AMblock_document
table. Question and details in thread.
TLDR: Why are there so many Local File System
records in block_document
table, even when I am not using Local File system directly on my code? Is it safe to just delete them?block_document
table. I did a quick analysis.
select bt.name,count(1) from block_document bd , block_type bt
where bt.id=bd.block_type_id
and bd.updated >= now() - interval '1 week'
group by bt.name;
Results are as follows:
"name","count"
Kubernetes Job,19
Azure,1
Date Time,11
Local File System,17072
NOTE: This is data for only 1 week. I decrypted the data
with the Fernet key. The decrypted values are the same and refer to the PREFECT_DIRECTORY base path.
This is pain ful as the table is estimated to be 50+ GB for us and most of the values are just encrypted duplicates of the same value.import prefect
from prefect import flow, task
@task
def ta():
prefect.get_run_logger().info("THis is from task")
@flow
def sub_flow():
prefect.get_run_logger().info("This is sub flow")
ta()
@flow
def main_flow():
prefect.get_run_logger().info("This is main flow")
sub_flow()
ta()
if __name__ == "__main__":
main_flow()
This created a single record in block document.
I took the dump sql of the entire database in text format, searched for the ID of the block document record and there are no references to that ID in any other tables.select bd.* from block_document bd
where bd.updated < now()-interval '1 week'
and is_anonymous = True
and exists (
select 1 from block_type bt where bt.id=bd.block_type_id
and bt.name='Local File System'
)
*delete instead of selectJake Kaplan
08/02/2023, 1:40 PMDeceivious
08/02/2023, 1:41 PM