[solved] When I create a flow that runs a namespaced job, prefect actually creates 2 jobs.

dummy

runs on the

aks-spot

node but

prefect-job

runs on the

aks-system

node (and I don’t want it running on the system node pool). Is there a way to configure tolerations and affinities for the

prefect-job

pod?

If you are running a Kubernetes agent, then each flow run is executed as a Kubernetes job (this should be the last pod I think). Then when you start a job using RunNamespacedJob task, it creates a separate job. So if you want to avoid spinning up 2 jobs and you can execute your logic within your Python flow definition, then you can get rid of the RunNamespacedJob task and run everything within the main flow run pod.

Ah okay. I would actually like the flow scheduling and business logic to be in separate repositories to 1. prevent sticky dependency situations 2. allow teams that use other languages to participate It looks like the documentation on a kubernetes run isn’t super clear on what it does. I’m not sure if I can specify a

job_template

and also pass in a

RunNamedspacedJob

task. Let me try doing that

it’s definitely doable. The job template is part of run config and is completely independent of what tasks you run, whether it’s dbt or Kubernetes tasks. The template only determines how your flow run gets deployed as a Kubernetes job.

flow.run_config = KubernetesRun(job_template_path="<s3://bucket/path/to/spec.yaml>")

Is the job template for kubernetes run supposed to be a fully fledged job template? Will the agent override some of the parameters and specify image, command, args etc ?

This is the default job_template - it’s not a fully fledged job definition, it’s just a template defining things like the container name (there must be a container named “flow” - this will be used for the flow run) + optionally you could set custom things like service accounts etc. Greetings from Berlin, too! 🇩🇪 (Gern geschehen!)

Das ist ja der Wahnsinn. Ich versuche das denn

If you want to submit a feature request, would you be willing to open a Github issue to discuss the problem and show why you can’t use the job template to customize the run config to your use case?

Already did 😄

That’s me

“If I specify a 

job_template

 then I cannot use the 

RunNamespacedJob

 task.” - can you explain why?

can you explain why?

I actually looked into it, and it was because I didn’t specify a container name. When I fixed that it actually worked! This is the code snippet if anyone happens across this

run_config_job_template = {
    "apiVersion": "batch/v1",
    "kind": "Job",
    "spec": {
        "template": {
            "spec": {
                "containers": [
                    {
                        "name": "flow"
                    }
                ],
                "affinity": {
                    "nodeAffinity": {
                        "requiredDuringSchedulingIgnoredDuringExecution": {
                            "nodeSelectorTerms": [
                                {
                                    "matchExpressions": [
                                        {
                                            "key": "agentpool",
                                            "operator": "In",
                                            "values": ["spotd2as"],
                                        }
                                    ]
                                }
                            ]
                        }
                    }
                },
                "tolerations": [
                    {
                        "key": "<http://kubernetes.azure.com/scalesetpriority|kubernetes.azure.com/scalesetpriority>",
                        "operator": "Equal",
                        "value": "spot",
                        "effect": "NoSchedule",
                    }
                ]
            }
        }
    }
}

k8s_run_config = KubernetesRun(job_template=run_config_job_template)

Nice work and thanks for sharing! Is it OK to close the Github issue then, or do you want to keep it?

I will post this to github, but I would still like the feature request to stay since it eliminates boilerplate

Ebenfalls! 🙂

Can you include this workaround / update your note on the

RunNamespacedJob

bit in your issue? I was quite confused by that part of your request.

shore