https://prefect.io logo
d

Deceivious

08/01/2023, 9:01 AM
Hello Prefect Devs, Is there any reason an on delete cascade delete foreign key relationship from
flow_run.id
to
log.flow_run_id
/
task_run.id
to
log.task_run_id
does not exist?
j

Jake Kaplan

08/01/2023, 1:02 PM
Theres not a major reason to enforce referential integrity there. For example there isn't a place to retrieve logs individually where it's harmful for their parent run to not exist. On the flip side it's also a big performance hit to enforce the FK on log insert and on also on run deletion via cascade, where every run could have N logs
d

Deceivious

08/01/2023, 1:18 PM
Thanks @Jake Kaplan We have a self hosted Prefect Database on prod and the logs table in prefect has grown to be larger than our actual production database. I was looking into solutions for this. Is there any plans to regarding this in Prefect OSS? Or its up to the users to manage this by themselves?
j

Jake Kaplan

08/01/2023, 1:43 PM
I think a couple people recently have asked about managing old data in Prefect OSS. At the moment it is up to the user's to manage themselves if hosting their own setup. However we would definitely welcome a contribution if you're interested? This is the only open issue for it I believe
d

Deceivious

08/01/2023, 2:34 PM
I opted to using
Copy code
ALTER TABLE public.log ADD CONSTRAINT fk_log_flow_run_id 
FOREIGN KEY (flow_run_id) REFERENCES flow_run(id) ON DELETE CASCADE;
We have different flow retention period based on the deployment; hence simple logic like
pg_cron
to clear logs does not work.