hey so thus far we are successfully running flows with our own base img with a size of 1.5 gb on eks with the kubernetes agent…. is there any way to speed up the pod spin up (we are using s3 storage stored as script for flows)

@Mike Lev I assume you are running AWS EKS with self-managed EC2 data plane? Since pulling the image from ECR is the most time-consuming process during the pod startup, you could specify the image pull policy to only pull the image if this image is not already present on the worker. You could specify this either in your Kubernetes job template, or directly on KubernetesRun:

KubernetesRun(labels=["your-agent-label"], image="ecr-url", image_pull_policy="IfNotPresent")

hey @Anna Geller currently we are running on fargate nodes so it will spin up a fargate worker and then spin down… wondering if

imagePullPolicy

is relevant here

correct, there is a very small chance that the second time you run your flow on Fargate that it will run on the same instance, so the image will still have to be pulled almost always. In general, with Fargate you need to consider some added latency needed to provision a micro VM for the container and to pull the image on this micro VM. There are two options you can consider: 1. You could add a self-managed always-on node to your cluster (and add it to a different namespace) for low-latency flows. I actually wrote about it a year go here. 2. You could add a nodegroup with cluster autoscaler to keep the benefit of scalability and low-maintenance https://eksctl.io/usage/autoscaling/