Hi team, I've got some questions about Kubernetes Workers, specifically their use of base job templates. I'm adapting the prefect-worker helm chart, which suggests configuring the worker with a base-job-template.json file. However, I understand that work pools have their own job templates, and that these must be actively updated using the

prefect work-pool update

command (it is not updated automatically, say the next time the worker restarts). I intend to implement custom, dynamic job templates, so I want to make sure that my job template is always up-to-date for a given helm release. Is there a recommended way to do this?

Hey @Billy McMonagle! Theoretically, it seems like this would work, but can you elaborate more on what you’re trying to accomplish with custom dynamic job templates? I’d like to understand more to see if there are alternative approaches that would work or if there’s an opportunity to improve the framework.

Thanks Alex. Sure, there are a couple main considerations: • I'm trying to make everything branch-based... my helm releases are named

app-branch

, and then

{{ HELM_RELEASE }}

is used in a bunch of places to facilitate this. • Trying to use external-secrets instead of the manually created K8s secrets recommended in the docs • Not sure if this matters, but using IAM Roles for Service Accounts.

Gotcha, which parts of the base job template do you update to incorporate those branch names? I’m guessing that it’s either the default values under

variables

or the values in

job_configuration

directly.

Probably

job_configuration

directly.

Makes sense. Do you have a sense of what your ideal interface would be for making portions of your base job template dynamic? I have some ideas, but would love to hear your thoughts

Ideally for me it would be as Kubernetes-y as possible... given that this is genuinely a template which needs to be populated at flow run time, I get that it is pretty tricky (esp to find something that works for a range of users). For comparison, in the app I'm migrating, we're doing this, which works fine:

with open(job_template_path, "r") as job_file:
    job_template = yaml.safe_load(
        "".join(job_file.readlines()).format(
            HELM_CHART=HELM_CHART, HELM_RELEASE=HELM_RELEASE
        )
    )
infrastructure = KubernetesJob(
    name=f"{HELM_RELEASE}-prefect-job",
    image=IMAGE_NAME,
    image_pull_policy="Always",
    service_account_name=HELM_RELEASE,
    job=job_template,
    env=environment_variables,
    finished_job_ttl=300,
    namespace="default",
    stream_output=False,
)
infrastructure.save(name=f"{HELM_RELEASE}-kubernetes-job", overwrite=True)

That said, I like the idea of workers, and I like configuring this stuff at the worker level, because it simplifies the creation of a bunch of individual flows that then don't need to worry about infra at all unless they need something custom. I'd love to see Kubernetes Custom Resource Definitions for prefect resources, but that may be overkill.

Do you think being able to use environment variables in the base job template would work for your use case? That way you could use your branch-based environment variables in the base job template and as you deploy new workers it would update based on your worker deployment.

Yeah, I do think that would cover the bulk of what I'm looking for.