Scott Zelenka
06/04/2020, 7:22 PMimagePullPolicy
when launching a Job through the Kubernetes Agent?
Since the image of the Flow to run will always have a tag for the specific version, this value could be changed from Always
to IfNotPresent
. This would decrease the startup time for Flows that get executed frequently, since there's a good chance the image will already exist on the K8 node where it's being executed.Dylan
06/04/2020, 7:48 PMprefect agent install kubernetes
command outputs a YAML file you can use for the agent’s k8s deployment. If you save that output to a file you can alter it. You should be able to change the imagePullPolicy
that’s set by default and then call kubectl apply -f
using the YAML fileScott Zelenka
06/04/2020, 7:53 PMJob
, not the Agent
. So when the Agent
goes to spawn a Job
, the imagePullPolicy
of that Job
is set to Always
, but I would like to change to IfNotPresent
Or maybe the Job
YAML is copied from the Agent
that spawned it? I'll give it a testDylan
06/04/2020, 7:54 PMScott Zelenka
06/04/2020, 8:36 PMjob_spec
file, but we're trying to get the Flow to start executing faster. In practice, the job_spec
spawns one Job, to then trigger another Job with the YAML specified, which is increasing the amount of startup time, not decreasing it. (:
This specific use case is where we integrated a case management system with Prefect Cloud, such that when an event happens to a case, it'll trigger the execution of a Flow
with some parameters about that case.
Since this is essentially an asynchronous workflow, we want the Flow
to execute as quickly as possible. There's a Task
in the Flow
to write back to the case which triggered it (which is an input Parameter
). So once that note shows up in the case, the person working the case knows it completed.
The same Flow
is executed each time the event happens on a case, but with different Parameters
. Which means with the imagePullPolicy
set to Always
, each time it's going to the Docker Repository to re-download the image to the K8 node before it can start executing. This download happens quick enough, but it's a waste of resources that we could eliminate by setting the imagePullPolicy
to IfNotPresent
on the first Job
that the Agent
spawns.
We have about 200 K8 nodes, so the chances the image is already on the node is pretty high (depending on the volume of case activity). Which could be higher if we could also set the nodeAffinity
in the same YAML file.Dylan
06/04/2020, 9:00 PMjob_spec.yml
passed to the agent or the ability to modify the default imagePullPolicy
in the default job_spec
would solve your issue.
Would you mind if I posted your last message in a new issue? This seems like something we could potentially elevate to an agent flag. I’ll reference the above issue as well