https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I see consistently between 10-20 seco...
# ask-marvin
l
@Marvin I see consistently between 10-20 seconds of delay before starting my event based flow in kubernetes. Why is this so slow? Logs:
Copy code
Worker 'KubernetesWorker 83df4330-3d54-48b4-b504-67742590f013' submitting flow run 'e8b33d23-bccc-4a88-bf72-5bf465e093ca'
10:48:30 AM
prefect.flow_runs.worker

Creating Kubernetes job...
10:48:30 AM
prefect.flow_runs.worker

Completed submission of flow run 'e8b33d23-bccc-4a88-bf72-5bf465e093ca'
10:48:30 AM
prefect.flow_runs.worker

Opening process...
10:48:50 AM
prefect.flow_runs.runner
 > Running set_working_directory step...
m
✅ thought for 261.1 seconds
Short answer: that 10–20s gap is Kubernetes time, not Prefect time. Your worker submits a Job immediately (104830). The “Opening process…” log only appears once the container is actually running and the Prefect runner starts (104850). The time in between is Kubernetes scheduling the pod and pulling/starting the image. What typically contributes to the gap - Pod scheduling: finding a node with resources and scheduling the pod. - Image pull and start-up: contacting the registry, checking the digest, pulling layers if not cached, unpacking, and starting the container. This is usually the biggest chunk. - Cluster specifics: cold nodes from autoscaling, Fargate cold starts, slow registry/network, admission controllers, large images, mounting volumes, etc. How to confirm where time is spent - Inspect pod events for a run:
Copy code
# Find the pod by flow run label
kubectl get pods -n <namespace> -l "<http://prefect.io/flow-run-id=<your-flow-run-id>|prefect.io/flow-run-id=<your-flow-run-id>>" -o wide
kubectl describe pod <pod-name> -n <namespace>
# Look for events like: Scheduled, Pulling, Pulled, Created, Started with timestamps
This will show exactly how long scheduling and image pulling took. Practical ways to reduce startup latency - Ensure the image is cached on nodes - Pre-pull your flow image on nodes (DaemonSet or a one-off warm-up) so the pod doesn’t pull at run time. - Use a registry mirror/local cache (e.g., ECR/EKS pull-through cache, local Registry mirror) to speed up first pulls. - Set the image pull policy appropriately - If you pre-pull images and control versions, use 
imagePullPolicy: IfNotPresent
(or even 
Never
if you guarantee the image exists on nodes). - In your Kubernetes work pool’s job template, set 
image_pull_policy
to 
IfNotPresent
and use unique tags for new images to avoid stale caches. - Reduce image size - Use a slim base image and only required dependencies (custom Prefect image) to reduce pull time. - Keep nodes warm and avoid scale-up delays - Maintain a small baseline of ready nodes; avoid cold-starting nodes on every event. - Ensure resource requests/limits match reality so pods aren’t stuck pending while the scheduler hunts for capacity. - Network/registry optimizations - Use VPC endpoints/private link to your registry where applicable. - Avoid rate limits by authenticating pulls and using regional registries. An alternative for near-instant starts - Run a long-lived Process worker inside Kubernetes. Your flows run as subprocesses in the already-running worker pod, so there’s no Job/Pod scheduling or image pull per flow. - Trade-offs: less isolation between flows, shared interpreter/resources, best for trusted/lightweight workloads and CI-style use cases. - Docs: Process worker Relevant docs - Run flows on Kubernetes - Work pools and Kubernetes options - prefect-kubernetes integration If you can share a bit more, I can give tailored steps: - Which cluster (EKS/GKE/AKS/on-prem)? Fargate involved? - What image/tag are you using and roughly how large is it? What’s your 
imagePullPolicy
? - Are your nodes autoscaled to zero or often cold?
l
@Marvin I have pre-pulled the image on the nodes, as well imagePullPolicy is set to IfNotPresent, there are 4 kubernetes nodes that are available and this is the pod description with some commented sensitive fields:
Copy code
Name:             perfect-hoatzin-bgq7k-lxlbm
Namespace:        prefect-server-eng
Priority:         0
Service Account:  default
Node:             ....
Start Time:       Tue, 16 Dec 2025 11:10:57 +0200
Labels:           <http://batch.kubernetes.io/controller-uid=68913602-4861-4dce-a4a1-effb6f04281d|batch.kubernetes.io/controller-uid=68913602-4861-4dce-a4a1-effb6f04281d>
                  <http://batch.kubernetes.io/job-name=perfect-hoatzin-bgq7k|batch.kubernetes.io/job-name=perfect-hoatzin-bgq7k>
                  controller-uid=68913602-4861-4dce-a4a1-effb6f04281d
                  job-name=perfect-hoatzin-bgq7k
                  <http://prefect.io/deployment-id=d188e870-c829-47c0-8f00-059bc0e05f36|prefect.io/deployment-id=d188e870-c829-47c0-8f00-059bc0e05f36>
                  <http://prefect.io/deployment-name=handle_freight_search|prefect.io/deployment-name=handle_freight_search>
                  <http://prefect.io/deployment-updated=2025-12-16T09-10-57.299736-00-00|prefect.io/deployment-updated=2025-12-16T09-10-57.299736-00-00>
                  <http://prefect.io/flow-id=1e836f15-3296-4f0f-bdbc-3d456e406701|prefect.io/flow-id=1e836f15-3296-4f0f-bdbc-3d456e406701>
                  <http://prefect.io/flow-name=timocom-search-flow|prefect.io/flow-name=timocom-search-flow>
                  <http://prefect.io/flow-run-id=cbd37995-129d-4923-b838-43ff00c9882c|prefect.io/flow-run-id=cbd37995-129d-4923-b838-43ff00c9882c>
                  <http://prefect.io/flow-run-name=perfect-hoatzin|prefect.io/flow-run-name=perfect-hoatzin>
                  <http://prefect.io/version=3.6.3|prefect.io/version=3.6.3>
                  <http://prefect.io/work-pool-id=c9d71eec-ccfd-4c80-a598-7dad62beab81|prefect.io/work-pool-id=c9d71eec-ccfd-4c80-a598-7dad62beab81>
                  <http://prefect.io/work-pool-name=k8s-pool|prefect.io/work-pool-name=k8s-pool>
                  <http://prefect.io/worker-name=KubernetesWorker-3e69e8d2-5563-4cf7-9248-a6da923727ed|prefect.io/worker-name=KubernetesWorker-3e69e8d2-5563-4cf7-9248-a6da923727ed>
Annotations:      <http://cni.projectcalico.org/containerID|cni.projectcalico.org/containerID>: 8b00d65fd1b0126cef480f327e37199581026b5f46d971b3dc1347fe95e27bfb
                  <http://cni.projectcalico.org/podIP|cni.projectcalico.org/podIP>: 
                  <http://cni.projectcalico.org/podIPs|cni.projectcalico.org/podIPs>: 
Status:           Succeeded
IP:               ...
IPs:
  IP:           ....
Controlled By:  Job/perfect-hoatzin-bgq7k
Containers:
  prefect-job:
    Container ID:  <containerd://a2e0778e368ce398db7194ede939dbe62e439caeff4cb09e36dbea17ececde0>3
    Image:         ..../direct-freight-search-k8s:0.0.4
    Image ID:      ...../direct-freight-search-k8s@sha256:0e82e815b3f869cddaffeb8b767639e30137dcae48dfdd61288e6accc0db08c5
    Port:          <none>
    Host Port:     <none>
    Args:
      prefect
      flow-run
      execute
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 16 Dec 2025 11:10:59 +0200
      Finished:     Tue, 16 Dec 2025 11:11:39 +0200
    Ready:          False
    Restart Count:  0
    Environment:
      PREFECT_DEBUG_MODE:                         False
      PREFECT_API_URL:                            ....
      PREFECT_API_AUTH_STRING:                    ....
      PREFECT_API_TLS_INSECURE_SKIP_VERIFY:       True
      PREFECT_API_ENABLE_HTTP2:                   True
      PREFECT_SERVER_EPHEMERAL_ENABLED:           True
      PREFECT_WORKER_QUERY_SECONDS:               1.0
      PREFECT_WORKER_PREFETCH_SECONDS:            2.0
      PREFECT__FLOW_RUN_ID:                       cbd37995-129d-4923-b838-43ff00c9882c
      PREFECT_FLOW_RUN_EXECUTE_SIGTERM_BEHAVIOR:  reschedule
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cdc8x (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  kube-api-access-cdc8x:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  3m17s  default-scheduler  Successfully assigned prefect-server-eng/perfect-hoatzin-bgq7k-lxlbm to ....
  Normal  Pulled     3m16s  kubelet            Container image "..../direct-freight-search-k8s:0.0.4" already present on machine
  Normal  Created    3m16s  kubelet            Created container: prefect-job
  Normal  Started    3m15s  kubelet            Started container prefect-job
m
✅ thought for 396.3 seconds
Thanks for the details — they’re super helpful. From the pod you shared: - Scheduled → Pulled → Created → Started all happened within ~2 seconds: - Start Time: 111057 - Container Started: 111059 - Event: “image already present on machine” confirms no pull time That means for this run Kubernetes startup was fast. The 10–20s gap you saw earlier (between “Completed submission” and “Opening process…”) would then be: - Either Kubernetes latency on other runs (scheduler queueing, admission webhooks, CNI init, etc.), or - A mismatch between which run you’re comparing (the example with 20s may be a different run than the one you described) A few things to check on a slow run to pinpoint where time is going - Describe the pod for that specific slow run and look at timestamped events:
Copy code
kubectl get pods -n <namespace> -l "<http://prefect.io/flow-run-id=<FLOW_RUN_ID>|prefect.io/flow-run-id=<FLOW_RUN_ID>>" -o name
kubectl describe pod <pod-name> -n <namespace>
- Pull precise timestamps (use these three to see where time accrues):
Copy code
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.metadata.creationTimestamp}{"\n"}{.status.startTime}{"\n"}{.status.containerStatuses[0].state.running.startedAt}{"\n"}'
Compare those to the worker log times for “Completed submission of flow run …”. If the gap is before pod creation/scheduling, look at: - Job controller and admission webhooks latency (OPA Gatekeeper/Kyverno, mutating/validating webhooks can add 5–10s). - CNI setup (Calico, etc.) on certain nodes. - API server throttling or network latency to the API server. - BestEffort QoS: consider adding small cpu/memory requests to avoid scheduling edge cases. What else could add time before “Running set_working_directory step…” - “Opening process…” is the first thing the Prefect runner logs inside the container. Any delay before that is 100% Kubernetes time. - After that, if you see additional startup time, it’s typically from deployment pull steps (e.g., installing packages via 
requires
, cloning code, resolving blocks/variables). You can avoid that by baking code/deps into the image and using module-path entrypoints. Ways to minimize startup for event-driven flows - Keep what you have (pre-pulled image + IfNotPresent) — that’s good. - Add minimal resource requests (e.g., 100m CPU, 128Mi memory) to avoid BestEffort and help the scheduler place pods promptly. - Check for slow webhooks/CNI by comparing “Scheduled” → “Started” across nodes. If you see a pattern on certain nodes, investigate those node components. - For near-instant starts, run a long-lived Process worker inside the cluster. Flows run as subprocesses in that pod, so there’s no Job/Pod spin-up or image pull per run. Trade-offs: shared interpreter and less isolation, best for trusted/lightweight event workloads. - Docs: Process worker Prefect docs for reference - Run flows on Kubernetes - Work pools and Kubernetes options - prefect-kubernetes integration If you can share the 
kubectl describe pod
events for one of the slow runs (the one that shows ~20s), I’ll help pinpoint exactly where the time is spent and suggest targeted fixes.
l
@Marvin Can you tell me what exactly happens between these two logs on a run? 
Completed submission of flow run '2e49864c-312f-498f-a88a-0e85795ef3f4'
10:09:37 AM
prefect.flow_runs.worker
 
Opening process...
10:10:14 AM
prefect.flow_runs.runner
m
✅ thought for 47.8 seconds
Short version: that entire gap is Kubernetes spinning up your Job pod and the container process reaching the Prefect runner. No Prefect code runs for your flow until the container starts, and the first thing Prefect logs inside the container is “Opening process…”. Detailed sequence between those two logs 1) Worker finished submitting the Job - Log: Completed submission of flow run … - Action: Prefect’s Kubernetes worker has posted a Job to the Kubernetes API and returns immediately. 2) Kubernetes control plane - The Job controller observes the new Job and creates a Pod for it. - Admission webhooks (mutating/validating) may run on the Pod and inject/validate config. - The scheduler assigns the Pod to a node. 3) On the selected node (kubelet / container runtime) - Ensures the image is available (IfNotPresent: verify digest; pull if missing). - Sets up volumes and mounts (service account token, ConfigMaps/Secrets, projected volumes). - Configures networking via your CNI (e.g., Calico), allocates IP, sets iptables/routes. - Creates and starts the container. 4) Container process starts and hands control to Prefect - The container’s command/args execute, typically:
Copy code
prefect flow-run execute
- Python interpreter starts; Prefect loads settings/env, parses CLI args. - Prefect initializes the Runner and logs: - Log: Opening process… - This is the parent runner about to spawn the subprocess that will load the deployment, run pull steps, and execute your flow. Key implications - Everything prior to “Opening process…” is outside Prefect’s control and owned by Kubernetes + your node runtime. - If you see 10–40s gaps, they usually come from scheduler queueing, admission webhook latency, CNI setup, and container startup (even with a pre-pulled image). - After “Opening process…”, Prefect begins running deployment pull steps (e.g., working directory, code/source fetch if configured, etc.), which is when you’ll see logs like “Running set_working_directory step…”. How to pinpoint the time distribution on a specific run - Describe the pod for that exact flow run and compare timestamps:
Copy code
kubectl get pods -n <ns> -l "<http://prefect.io/flow-run-id=<FLOW_RUN_ID>|prefect.io/flow-run-id=<FLOW_RUN_ID>>" -o name
  kubectl describe pod <pod-name> -n <ns>
  # Pull precise timestamps
  kubectl get pod <pod-name> -n <ns> -o jsonpath='{.metadata.creationTimestamp}{"\n"}{.status.startTime}{"\n"}{.status.containerStatuses[0].state.running.startedAt}{"\n"}'
- These will show when the Pod was created, when it started running on the node, and when the container process actually started (which precedes the “Opening process…” log by a fraction of a second). Relevant Prefect docs - Run flows on Kubernetes - Work pools and Kubernetes options - prefect-kubernetes integration If you share the 
describe pod
events and those three timestamps for a slow run, I can map each segment of the 37s precisely to Kubernetes stages vs. container start-up.
13 Views
Open in Slack
PreviousNext
Powered by