Hi! The folder in .prefect is steadily growing in size - is this the default space for the database ...

Stefan

03/02/2023, 9:59 AM

Hi! The folder in .prefect is steadily growing in size - is this the default space for the database or is it some temp files that are not deleted?

✅ 1

Tim-Oliver

03/02/2023, 10:00 AM

storage

is where prefect caches your task-run results by default.

Stefan

03/02/2023, 10:02 AM

Oh. For representation in the UI or API, or for what purpose? They are wildly varying in size.

Tim-Oliver

03/02/2023, 10:35 AM

https://docs.prefect.io/concepts/results/#persisting-results

Tim-Oliver

03/02/2023, 10:35 AM

A task is only executed if the result does not exist yet.

Tim-Oliver

03/02/2023, 10:36 AM

If you run the same flow twice with the same parameters and with persisting results on the second run will use the cached results.

Tim-Oliver

03/02/2023, 10:37 AM

By default prefect uses pickle to save the returned values of a task as binary blob. The size of the files you see in

storage

depends on the size of the objects returned by your tasks.

Stefan

03/02/2023, 11:42 AM

OK I see. It's the storage of retries, results etc. How long are they persisted? Should we clean the folder?

Tim Galvin

03/02/2023, 2:41 PM

Is the persistent cache the default behaviour? In other words, are these binary blobs containing task run results unless told otherwise?

Christopher Boyd

03/02/2023, 3:20 PM

https://docs.prefect.io/concepts/results/#toggling-persistence

Tim Galvin

03/03/2023, 1:36 AM

Perfect! Thanks @Christopher Boyd

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.