jars

    jars

    2 years ago
    We noticed after upgrading our GKE agent from 0.11 to 0.13, GKE does not keep our Job history under GKE "Workloads" tab. It used to contain the agent Deployment, and all jobs, but now jobs are automatically getting cleaned-up. Is this expected, and follow-up: is it configurable?
    m

    Marwan Sarieddine

    2 years ago
    I think this is due to the resource-manager being deployed with the agent now - which clears the jobs/resources after a run
    nicholas

    nicholas

    2 years ago
    Hi @jars - is there a reason you'd want to disable automatic clean up of job history? In our experience, not running the resource manager incurred some non-trivial costs in k8s.
    jars

    jars

    2 years ago
    @nicholas we think the garbage collection is awesome, but having it more configurable would also be great. Being able to specify the last n days or weeks of history to keep, for example. Sometimes we want to inspect a completed job's container logs. But since the job itself is cleaned up, accessing the StackDriver logs is a bit cumbersome. We haven't had a Flow failure yet in our 0.13 deployment, so not sure what the default behaviour is here, but if it also cleans up the job history, this would be undesirable for us, as we want to inspect what went wrong.
    nicholas

    nicholas

    2 years ago
    Gotcha @jars - I think that makes sense. Would you mind opening a feature ticket for that with what you'd like to see?
    jars

    jars

    2 years ago
    Can we use Marvin for that? I see ppl using that around here ๐Ÿ˜›
    nicholas

    nicholas

    2 years ago
    Sure thing!
    @Marvin open "Provide Agent configuration options for k8s resource manager"
    Marvin

    Marvin

    2 years ago
    jars

    jars

    2 years ago
    So cool!
    nicholas

    nicholas

    2 years ago
    you hear that @Marvin? someone thinks you're cool
    (marvin is currently ignoring prefect employees cause of the 5k contest)
    jars

    jars

    2 years ago
    hehehe
    I'll tel him then
    @Marvin, you're pretty cool.
    Marvin

    Marvin

    2 years ago
    Don't feel you have to take any notice of me, please. I'll just enter you in the contest @jars.
    jars

    jars

    2 years ago
    nice
    nicholas

    nicholas

    2 years ago
    Our friendliest coworker
    jars

    jars

    2 years ago
    they say it's all in the training ๐Ÿ˜›
    h

    Hui Zheng

    2 years ago
    lol
    thatโ€™s so cool. @nicholas and @Marwan Sarieddine could you also share more about the
    resource-manager
    ? Is there a doc or reference for us to understand how it works? thank you
    nicholas

    nicholas

    2 years ago
    Hi @Hui Zheng! You can read more about resource manager tasks and the k8s agent in the API docs here: https://docs.prefect.io/api/latest/agent/kubernetes.html#kubernetesagent and here: https://docs.prefect.io/api/latest/tasks/resources.html#resourcemanager