https://prefect.io logo
Title
s

Scott Zelenka

06/04/2020, 7:22 PM
Is there an easy way to alter the
imagePullPolicy
when launching a Job through the Kubernetes Agent? Since the image of the Flow to run will always have a tag for the specific version, this value could be changed from
Always
to
IfNotPresent
. This would decrease the startup time for Flows that get executed frequently, since there's a good chance the image will already exist on the K8 node where it's being executed.
đź‘€ 1
d

Dylan

06/04/2020, 7:48 PM
Hi @Scott Zelenka! The
prefect agent install kubernetes
command outputs a YAML file you can use for the agent’s k8s deployment. If you save that output to a file you can alter it. You should be able to change the
imagePullPolicy
that’s set by default and then call
kubectl apply -f
using the YAML file
Let me know if that works!
s

Scott Zelenka

06/04/2020, 7:53 PM
I'm looking to modify the
Job
, not the
Agent
. So when the
Agent
goes to spawn a
Job
, the
imagePullPolicy
of that
Job
is set to
Always
, but I would like to change to
IfNotPresent
Or maybe the
Job
YAML is copied from the
Agent
that spawned it? I'll give it a test
d

Dylan

06/04/2020, 7:54 PM
Ah my apologies
In that case you’ll need to specify your own job spec yaml
One moment
We have an issue open on this topic: https://github.com/PrefectHQ/prefect/issues/1658
Would you mind commenting with your use case?
It is possible to take control over your k8s job spec using the KubernetesJobEnvironment: https://docs.prefect.io/orchestration/execution/k8s_job_environment.html#initialization
s

Scott Zelenka

06/04/2020, 8:36 PM
We know about the
job_spec
file, but we're trying to get the Flow to start executing faster. In practice, the
job_spec
spawns one Job, to then trigger another Job with the YAML specified, which is increasing the amount of startup time, not decreasing it. (: This specific use case is where we integrated a case management system with Prefect Cloud, such that when an event happens to a case, it'll trigger the execution of a
Flow
with some parameters about that case. Since this is essentially an asynchronous workflow, we want the
Flow
to execute as quickly as possible. There's a
Task
in the
Flow
to write back to the case which triggered it (which is an input
Parameter
). So once that note shows up in the case, the person working the case knows it completed. The same
Flow
is executed each time the event happens on a case, but with different
Parameters
. Which means with the
imagePullPolicy
set to
Always
, each time it's going to the Docker Repository to re-download the image to the K8 node before it can start executing. This download happens quick enough, but it's a waste of resources that we could eliminate by setting the
imagePullPolicy
to
IfNotPresent
on the first
Job
that the
Agent
spawns. We have about 200 K8 nodes, so the chances the image is already on the node is pretty high (depending on the volume of case activity). Which could be higher if we could also set the
nodeAffinity
in the same YAML file.
d

Dylan

06/04/2020, 9:00 PM
@Scott Zelenka Right, I think I understand. Given your usecase, I think the ability to specify the default
job_spec.yml
passed to the agent or the ability to modify the default
imagePullPolicy
in the default
job_spec
would solve your issue. Would you mind if I posted your last message in a new issue? This seems like something we could potentially elevate to an agent flag. I’ll reference the above issue as well