Hi all We ve recently noticed our prefect postgres database Prefect Community #ask-community

Hi all! We've recently noticed our prefect postgre...

Samuel Hinton

10/02/2023, 10:24 AM

Hi all! We've recently noticed our prefect postgres database has hit 20GB in size, which seems like a large size. Does anyone know if prefect is saving out task results / artifacts (note I have no explicit artifacts in any of my flows), and if there are ways to configure a retention policy (like to delete things after a week)?

👀 1

Nate

10/02/2023, 10:38 PM

hi @Samuel Hinton - are you below prefect 2.13.1? this PR fixed a problem (released in 2.13.1) where we were unnecessarily saving anonymous blocks, which others have noted contributed to unanticipated db growth

Samuel Hinton

10/03/2023, 12:31 AM

I will check our prefect version now!

👍 1

Samuel Hinton

10/03/2023, 1:33 AM

@Nate so it looks like we're using prefect 2.11.2, but upgrading to 2.13.4 is causing all sorts of havoc with the db. I can't even get a

prefect server database reset -y

to run properly. Is there another command even more fundamental that database reset to just wipe the whole thing, or should I get infra on the line when they wake up to do hard wipe?

Samuel Hinton

10/03/2023, 1:34 AM

Getting things like

DETAIL: Key (deployment_id)=(09085e52-ed16-4832-9061-7e161b037189) is not present in table "deployment".

and

insert or update on table "flow_run" violates foreign key constraint

When trying to run the database reset.

Samuel Hinton

10/03/2023, 1:35 AM

I wonder if this has anything to do with a change from

PREFECT_ORION_DATABASE_CONNECTION_URL

PREFECT_API_DATABASE_CONNECTION_URL

that I had to do?

Nate

10/03/2023, 2:35 AM

dang, sorry that's happening! it feels likely that there would be issues if you are still using the removed deprecated settings on 2.13.4 - all

PREFECT_ORION_*

settings should be removed as of 2.13.0 - following their deprecation period as linked, there's the

prefect config validate

(for your current profile) to switch those settings names, but I imagine you might have other places to update those settings in CI etc

Is there another command even more fundamental that database reset to just wipe the whole thing, or should I get infra on the line when they wake up to do hard wipe?

unfortunately, im not aware of other reset options, though i may not be the best resource on this exact topic - i can ask around as soon as i can tomorrow if that'd be helpful

Samuel Hinton

10/03/2023, 2:40 AM

Its all good, once infra in London is awake I'll ask them to clear it out. Will comment here if I manage to fix things up 🙂

👍 1

Deceivious

10/04/2023, 9:09 AM

@Samuel Hinton There are lots of stuff that Prefect does not manually clear up. Ive outlined some of them here in this issues https://github.com/PrefectHQ/prefect/issues/10000 Usually its the logs that takes up space and the duplicated infra blocks on every flow run [which has been fixed]. Hoping prefect will fix the issues Ive mentioned in the ticket and support a long running server instead of a server that needs to be managed manually.

Deceivious

10/04/2023, 9:09 AM

++ The title of the issues has been edited but its more than just the logs.

Samuel Hinton

10/04/2023, 9:37 AM

Ah this is exactly what I was seeing when trying to do the reset. I'm confused why the reset approach is "downgrade" and not just "nuke"

63 Views

Open in Slack

Previous Next