https://prefect.io logo
#prefect-server
Title
# prefect-server
s

Simon Gasse

10/26/2021, 11:49 AM
Hi! We are using Prefect with an S3 backend for a few month now and have accumulated >100GB in
.prefect_result
files. Is there an automated way in Prefect to clear all data older than a certain date, or do we have to rely on S3 for that?
a

Anna Geller

10/26/2021, 11:58 AM
@Simon Gasse the only way I can think of that Prefect would remove this old data for you would be if you build (and run on schedule) a custom Flow that lists the S3 objects in this bucket, looks at the
lastModified
timestamp and based on that, manually removes the old objects. I honestly believe that using S3 lifecycle policy is the easiest and most reliable way of regularly purging old result files. If you don’t need those results in general, you can turn off checkpointing on a task:
Copy code
@task(checkpoint=False)
def ...
k

Kevin Kho

10/26/2021, 1:55 PM
Hey @Simon Gasse, aside from turning checkpoint to False, you can also specify the result filename to make it just overwrite the same files so you always have the last set of results.
upvote 1
s

Simon Gasse

10/26/2021, 3:15 PM
Alright, thank you @Anna Geller and @Kevin Kho for the quick feedback and your suggestions! 🙂 • Which functionality/logs are we losing in the UI if we deactivate the checkpointing? • @Kevin Kho could you point me to some documentation to name the results?
k

Kevin Kho

10/26/2021, 3:20 PM
You don’t lose logging functionality. You lose the ability to retry from failure. This will be what you’re looking for
🙏 1
s

Simon Gasse

10/27/2021, 7:35 AM
Oh OK, thank you very much! 🙂 👍