when running prefect jobs on a kubernetes cluster, is it possible to send out a memory request upon flow run creation? currently right now I can set a memory request via the config of the worker but this sets the limit for all jobs, I want to override this value with a value set at job creation
p
Patrick Alves
11/26/2024, 4:54 PM
I am trying to do something similar, lmk if you make it work š
š 1
n
Nate
11/26/2024, 4:59 PM
yes! there are two steps here
⢠customize your work pools job template to hoist up
mem_request
and
cpu_request
into the
variables
section
⢠use per deployment or per flow run
job_variables
overrides that customize those values
on the first point, we should add a guide but essentially if you go into the Advanced tab of your work pools base job config, it will hopefully make sense what I mean
w
will
11/26/2024, 5:01 PM
thank you this is helpful, I already have the first point done to send in the request baseline for each job, sending in the job variable on a job basis was what I was looking for!
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.