Alexis Lucido

    Alexis Lucido

    8 months ago
    Hi all, and all the best for the year to come. My Prefect processes running locally generated too many logs, filling up the memory of my on-premise virtual machine. Our Ops team increased the size of the VM memory and I have disabled checkpointing, however many late runs are registered (several thousands). I want to delete these late runs, but I guess I need to launch our agent (a local-type one) that tries to catch up with all the late runs I am trying to delete through the UI. Thus, the process of deleting the late runs is very slow, i.e. it deletes a few of them per minute. I add that we persist the backend database to keep an audit trace, or at least the best one we could have. Is it possible to delete the late runs without lauching the agent / trying to catch up with the old ones? Thank you very much in advance. Alexis
    Kevin Kho

    Kevin Kho

    8 months ago
    Hey @Alexis Lucido, a lot of server clear the logs at a regular interval to free up space. You can delete all the late runs in the Dashboard. What Prefect version are you on?
    Check this
    Alexis Lucido

    Alexis Lucido

    8 months ago
    Hi @Kevin Kho. Does that mean that I should disable checkpointing plus deleting server and supervisord (I used supervisord for deploying the agent) logs? In that case, only the Postgre database will be growing over time, right ? Is there some logs that I am missing here?
    Two other questions: What would be the cleanest way to delete these logs: a Prefect Flow or, say, a cron job? Finally, is the Posgre backend database stored in the Cloud if the Cloud option is preferred? Thanks in advance for answering these newbie questions
    PS: My prefect version is 0.5.10
    Kevin Kho

    Kevin Kho

    8 months ago
    5.10!? is it really 5 and not 15?
    Alexis Lucido

    Alexis Lucido

    8 months ago
    My bad it was a typo: indeed the version is 0.15.10
    Thank you in advance, and sorry for my late answer!
    Kevin Kho

    Kevin Kho

    8 months ago
    Ah ok lol. That would have been the oldest version I have ever seen here. First is either a flow or cron job to delete the logs. We host the database on Cloud yep
    But logs are limited to 2-3 week retention or 6 months for enterprise
    Yes disable checkpointing and only the db will grow over time
    Alexis Lucido

    Alexis Lucido

    8 months ago
    Alright thanks Kevin. I have been in contact with one of your commercial and he told me you would be able to increase logs retention in database because we operate in a domain with strict regulations and an audit trail is really important
    Following on the subject, we have seen a gradual increase of RAM usage over time on our VMs, mostly due to cached memory. Is it possible that it is caused by the backend database in the postgresql container? We have used the default settings for the core server and we persist the database: prefect server start --use-volume
    Kevin Kho

    Kevin Kho

    8 months ago
    That might indeed be the database. It is common for users to truncate the logs table and the task run table regularly because they grow very fast