Hi guys, I have been wondering about k8s and job executions, since it's widely known that Kubernetes cron jobs can create none or two executions of its schedule. My concern is of course that if something out of the ordinary happens in our cluster and our job either don't run or runs twice instead of the single run we would like to have. What guarantees is there for a flow when using the k8s agent that a job only runs once? Another concern is if we use CreateNamespacedJob from the task library, wouldn't that have the same problems as k8s cron jobs?

Its not a cron just a job

I am not that strong in k8s, but I was just thinking what guarantee is there that when spawning the job/pod that it only spawns once. If there is a network issue or what ever, couldn't that also happen for the k8s agent, that it didn't spawn the job probably? I was trying to look through the docs but I couldn't really find some clarification on it.

Hi @Dave CronJobs will not be scheduled if they are past their expected scheduled time. K8s will mark them as "missed". Prefect Agents (and presumably a flow calling

CreateNamespacedJob

) does not check this and will still run the flow/job as expected. In this sense, you shouldn't encounter a case where a job is not run. They do not explain the case where multiple jobs are created, I presume this would be due to a race condition between multiple CronControllers. We do not recommend running more than a single agent in a cluster (because it does not improve performance usually) so this shouldn't occur. Even with multiple agents running, it is unlikely that they are started at the exact same time and would have differing clocks.

Cool! We only have one agent per context/environment, so only one in k8s.

Hi Michael, I think our concern is not so much about the agent triggering no or multiple flows - that seems to be handled, since a flow needs to be registered to an agent - but rather if that flow pod uses a Kubernetes Task to e.g. run a CreateNamespacedJob task. The way that is done seems to be by invoking the Kubernetes API underneath using the standard Kubernetes python client and would that link not have the same vulnerabilities as a cron job at creating jobs or a job creating a pod?

Hey @Karsten Fyhn - from what I can tell the limitations are for

CronJobs

which uses a

CronManager

to create

Jobs

. The

Jobs

API will always succeed in creating a single job. The concern then would be that the job could create multiple pods (which should only ever happen in the failure case), see https://kubernetes.io/docs/concepts/workloads/controllers/job/#handling-pod-and-container-failures for details on that.