We noticed after upgrading our GKE agent from 0.11 to 0.13, GKE does not keep our Job history under GKE "Workloads" tab. It used to contain the agent Deployment, and all jobs, but now jobs are automatically getting cleaned-up. Is this expected, and follow-up: is it configurable?

I think this is due to the resource-manager being deployed with the agent now - which clears the jobs/resources after a run

Hi @jars - is there a reason you'd want to disable automatic clean up of job history? In our experience, not running the resource manager incurred some non-trivial costs in k8s.

@nicholas we think the garbage collection is awesome, but having it more configurable would also be great. Being able to specify the last n days or weeks of history to keep, for example. Sometimes we want to inspect a completed job's container logs. But since the job itself is cleaned up, accessing the StackDriver logs is a bit cumbersome. We haven't had a Flow failure yet in our 0.13 deployment, so not sure what the default behaviour is here, but if it also cleans up the job history, this would be undesirable for us, as we want to inspect what went wrong.

Gotcha @jars - I think that makes sense. Would you mind opening a feature ticket for that with what you'd like to see?

Can we use Marvin for that? I see ppl using that around here 😛

Sure thing!

So cool!

you hear that @Marvin? someone thinks you're cool

hehehe

Don't feel you have to take any notice of me, please. I'll just enter you in the contest @jars.

nice

Our friendliest coworker

they say it's all in the training 😛

lol

Hi @Hui Zheng! You can read more about resource manager tasks and the k8s agent in the API docs here: https://docs.prefect.io/api/latest/agent/kubernetes.html#kubernetesagent and here: https://docs.prefect.io/api/latest/tasks/resources.html#resourcemanager