https://prefect.io logo
Join Slack
Powered by
Hello team, I'm trying to migrate to prefect worke...
# prefect-cloud
k
Hello team, I'm trying to migrate to prefect worker from prefect agent, and there are some things I don't get. I have kuberetes job manifest specified inside of 
--base-job-template
during startup of prefect worker. And it's working fine, I'm able to create deployment and it runs just fine. But there is a scenario where I would like to update kubernetes job manifest - what should I do? This is what I do: 1. Update prefect worker configuration and deploy it to my Kubernetes cluster with new 
--base-job-template
2. Because work pool already exists, it's not updated. So I'm removing work pool and then worker creates new one with updated configuration. 3. Point 2 caused all deployments to lose work pool, and none of the execution is scheduled, it's in 
Late
state. The only way is to redploy it again, and it's problematic with 1000+ deployments. Is it supposed to work like that? Are there any better ways to update base-job-template?
✅ 1
o
I’ve just used the python cli to update the base job template
k
Ok, but I want to keep everything as a code and I would like to deploy newer prefect worker version. I know that I can update work pool manually or via cli, but is it really the only option?
o
Yeah, I’d like to know a better option as well. We have the python cli in CICD pipeline to update the base job template every time we deploy a flow to a work pool but it’s not optimal
We’d like to do it exactly as you’re asking 🙂
k
ahh, that's a pity. Thanks for the information, maybe someone from Prefect will see it and can implement some gitops way of handling it. Right now manual script seems like a workaround
@Olli Kavén maybe you had similar use-case and figured it out somehow. In most case the only update I need in kubernetes job manifest for prefect worker will be image tag we are using to run it. Is there any way to pass that information from different place, like from deployment? Then in 
base-job-template
I could specify some non-existant tag, and real tag would be overwritten by deployment. Or you are just handling it with your CI pipeline?
k
you can override the image name on your deployments rather than the template on the work pool
upvote 2
in fact, unless you're adding more fields to the work pool's template, like node selectors or volume mounts or something like that, I would expect that your work pool's base template should change very infrequently
upvote 2
the point of all the 
"{{ jinja_template_fields }}"
is that you can set them through a deployment's 
job_variables
so they don't have to change on the work pool all the time
upvote 2
k
thank you @Kevin Grismore, I missed that part but this is exactly what I was looking for! I tested it and it's working fine. Still I don't like the fact that I need to update all my deployments when I need to update baseJobManifest, but with properties need for it won't be something I will perform regularly. And maybe I will just update workPool manually in such cases together with code instead of doing change only in code and then removing workPool
o
@Karol Wolski sorry for late response but just to confirm that we are doing exactly as @Kevin Grismore suggested
👍 1
2 Views
Open in Slack
PreviousNext
Powered by