https://prefect.io logo
#prefect-community
Title
# prefect-community
j

jars

09/18/2020, 8:12 PM
We noticed after upgrading our GKE agent from 0.11 to 0.13, GKE does not keep our Job history under GKE "Workloads" tab. It used to contain the agent Deployment, and all jobs, but now jobs are automatically getting cleaned-up. Is this expected, and follow-up: is it configurable?
m

Marwan Sarieddine

09/18/2020, 8:17 PM
I think this is due to the resource-manager being deployed with the agent now - which clears the jobs/resources after a run
n

nicholas

09/18/2020, 8:51 PM
Hi @jars - is there a reason you'd want to disable automatic clean up of job history? In our experience, not running the resource manager incurred some non-trivial costs in k8s.
j

jars

09/18/2020, 9:54 PM
@nicholas we think the garbage collection is awesome, but having it more configurable would also be great. Being able to specify the last n days or weeks of history to keep, for example. Sometimes we want to inspect a completed job's container logs. But since the job itself is cleaned up, accessing the StackDriver logs is a bit cumbersome. We haven't had a Flow failure yet in our 0.13 deployment, so not sure what the default behaviour is here, but if it also cleans up the job history, this would be undesirable for us, as we want to inspect what went wrong.
n

nicholas

09/18/2020, 9:55 PM
Gotcha @jars - I think that makes sense. Would you mind opening a feature ticket for that with what you'd like to see?
j

jars

09/18/2020, 9:56 PM
Can we use Marvin for that? I see ppl using that around here ๐Ÿ˜›
n

nicholas

09/18/2020, 10:05 PM
Sure thing!
@Marvin open "Provide Agent configuration options for k8s resource manager"
j

jars

09/18/2020, 10:07 PM
So cool!
n

nicholas

09/18/2020, 10:10 PM
you hear that @Marvin? someone thinks you're cool
(marvin is currently ignoring prefect employees cause of the 5k contest)
j

jars

09/18/2020, 10:27 PM
hehehe
I'll tel him then
@Marvin, you're pretty cool.
m

Marvin

09/18/2020, 10:27 PM
Don't feel you have to take any notice of me, please. I'll just enter you in the contest @jars.
๐Ÿ˜† 1
j

jars

09/18/2020, 10:28 PM
nice
n

nicholas

09/18/2020, 10:28 PM
Our friendliest coworker
j

jars

09/18/2020, 10:29 PM
they say it's all in the training ๐Ÿ˜›
๐Ÿ˜† 1
h

Hui Zheng

09/18/2020, 11:26 PM
lol
thatโ€™s so cool. @nicholas and @Marwan Sarieddine could you also share more about the
resource-manager
? Is there a doc or reference for us to understand how it works? thank you
n

nicholas

09/18/2020, 11:31 PM
Hi @Hui Zheng! You can read more about resource manager tasks and the k8s agent in the API docs here: https://docs.prefect.io/api/latest/agent/kubernetes.html#kubernetesagent and here: https://docs.prefect.io/api/latest/tasks/resources.html#resourcemanager