Question regarding prefect server, we are running it in an EC2 Ubuntu instance with 20GB and getting close to using 100% of storage usage, is the server/docker containers storing log files? If so, where can I find out where?
k
Kevin Kho
05/27/2022, 5:54 PM
Two things to check:
1. the
.prefect
folder. task results are checkpointing by default
2. the database. the log table tends to be the biggest followed by task runs
j
Juan Valle
05/27/2022, 6:04 PM
The database is external and the .prefect folder doesn't have anything in it.
k
Kevin Kho
05/27/2022, 6:08 PM
Do you have configured Results for your tasks?
j
Juan Valle
05/27/2022, 6:10 PM
No, I do not.
k
Kevin Kho
05/27/2022, 6:12 PM
I guess it’s the Docker instances holding old images then? You can prune the ones you don’t need
j
Juan Valle
05/27/2022, 6:24 PM
Seems like we only have the images that are needed. I'll go ahead and look more into the Docker side of this issue. Thanks
m
Michael Moscater
08/03/2022, 7:56 PM
as a follow up for resolving this issue, it was in fact, the docker logs. There are multiple ways of solving this, best being rotating logs and archiving/deleting after set period
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.