https://prefect.io logo
Title
d

Daniel Komisar

12/17/2021, 1:27 PM
Hello everyone, is there any information on the data retention policy for prefect server? Thank you.
a

Anna Geller

12/17/2021, 1:35 PM
@Daniel Komisar what do you mean by data retention policy? What data specifically are you referring to? In general, Prefect operates on a hybrid execution model, which means that Prefect never receives your code or data, instead it operates on metadata. When you use Server, it means self-hosting so you decide on your own where and for how long you keep data such as logs. When talking about Prefect Cloud, we do have configurable log retention based on your Prefect Cloud plan. Does this answer your question?
d

Daniel Komisar

12/17/2021, 1:40 PM
Yes thank you.
🙌 1
k

Kevin Kho

12/17/2021, 2:54 PM
A lot of Server users find that they need to maintain the database by clearing a log table and task run table a some regular basis (month, 2 months). There are no retention limits for Server
d

Daniel Komisar

12/17/2021, 2:56 PM
When you say “maintain the database”, what happens if this maintenance isn’t done? Is it just for our own benefit, or will queries slow down over time if we don’t?
k

Kevin Kho

12/17/2021, 2:57 PM
Your API may fail because there’s no more memory to write new stuff in the database
d

Daniel Komisar

12/17/2021, 2:58 PM
Ah okay this sounds important, I have a couple questions. • How can well tell this is about to happen? • Any idea how much data it can hold? • Can you clear it out automatically for us?
k

Kevin Kho

12/17/2021, 3:01 PM
We would have no idea on your infrastructure. It would be when the machine that holds the database runs out of memory. We don’t take responsibility for maintaining server deployments.
d

Daniel Komisar

12/17/2021, 3:01 PM
To give a little more context, we don’t have a very flow heavy workload. The only intensive part is that flows generate somewhat large logs. So I think this may not be a problem for us for a while, but it does sound like we will need to address it at some point and I’m not sure when that is.
Sorry if I was unclear, this is all on Prefect Cloud.
k

Kevin Kho

12/17/2021, 3:02 PM
Ah Cloud we have the 2 week or 3 week retention period. Enterprise gives 6 months though so stuff before that retention gets cleared. If you need persistence of logs more than that, some users ship the logs to DataDog
d

Daniel Komisar

12/17/2021, 3:03 PM
Thank you, understood.
👍 1