Hi! We are using Prefect with an S3 backend for a few month now and have accumulated >100GB in

.prefect_result

files. Is there an automated way in Prefect to clear all data older than a certain date, or do we have to rely on S3 for that?

@Simon Gasse the only way I can think of that Prefect would remove this old data for you would be if you build (and run on schedule) a custom Flow that lists the S3 objects in this bucket, looks at the

lastModified

timestamp and based on that, manually removes the old objects. I honestly believe that using S3 lifecycle policy is the easiest and most reliable way of regularly purging old result files. If you don’t need those results in general, you can turn off checkpointing on a task:

@task(checkpoint=False)
def ...

Hey @Simon Gasse, aside from turning checkpoint to False, you can also specify the result filename to make it just overwrite the same files so you always have the last set of results.

Alright, thank you @Anna Geller and @Kevin Kho for the quick feedback and your suggestions! 🙂 • Which functionality/logs are we losing in the UI if we deactivate the checkpointing? • @Kevin Kho could you point me to some documentation to name the results?

You don’t lose logging functionality. You lose the ability to retry from failure. This will be what you’re looking for

Oh OK, thank you very much! 🙂 👍