https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I don't see any events in my new kube...
# marvin-ai
l
@Marvin I don't see any events in my new kubernetes Prefect server, even when i manually send event to the server. Simple flows run but automations for sending slack alerts on failure don't. What could the issue be?
m
✅ thought for 213.3 seconds
Short answer: in Kubernetes, “no events and no automations” almost always means the Prefect background services aren’t running and/or the messaging layer isn’t configured for multi-pod. Simple flows can still run because the API is up, but events and automations rely on services communicating via a broker. Likely causes (Prefect 3.x): - Background services not running: only 
prefect server start
is running, but not 
prefect server services start
. - In-memory messaging in a multi-pod setup: defaults work locally, but in Kubernetes you need Redis so services and API can share events. - Automations services disabled: triggers/actions not enabled (rare; they default to on). - WebSocket stream disabled: UI won’t show events in real-time if streaming is off (events may still be stored). What to check and fix 1) Ensure services are running in exactly one pod - API pods (can scale):
Copy code
prefect server start --host 0.0.0.0 --no-services
- Background services (replicas must be 1):
Copy code
prefect server services start
CLI help for services:
Copy code
prefect server services start --help
2) Configure Redis messaging (critical in Kubernetes) Set env vars so all server/service pods share the same broker:
Copy code
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_REDIS_MESSAGING_HOST=<your-redis-host>
PREFECT_REDIS_MESSAGING_PORT=6379
# optional auth
PREFECT_REDIS_MESSAGING_USERNAME=<user>
PREFECT_REDIS_MESSAGING_PASSWORD=<password>

# recommended for event ordering across pods
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
3) Ensure event/automation services aren’t disabled These default to true, but verify:
Copy code
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true
PREFECT_API_EVENTS_STREAM_OUT_ENABLED=true
4) Quick smoke tests - API ready:
Copy code
curl -sS http://<server-host>:4200/api/ready
- Send a test event (204 expected):
Copy code
curl -X POST http://<server-host>:4200/api/events \
  -H "Content-Type: application/json" \
  -d '[{"event":"test.event","resource":{"prefect.resource.id":"test"},"occurred":"2025-01-01T00:00:00Z"}]'
- See if it was persisted:
Copy code
curl -X POST http://<server-host>:4200/api/events/filter \
  -H "Content-Type: application/json" -d '{"filter": {}, "limit": 10}'
Minimal K8s pattern - One deployment for API:
Copy code
command: ["prefect", "server", "start", "--host", "0.0.0.0", "--no-services"]
- One deployment for services (replicas=1):
Copy code
command: ["prefect", "server", "services", "start"]
- Both share the same env vars for Redis and DB. If you’re using Helm, the official chart supports this separation and Redis setup. See Prefect Helm chart. A few questions to pinpoint your issue: - How did you deploy the server (Helm vs custom manifests)? - Do you have a separate “services” pod running 
prefect server services start
? If so, is it scaled to 1? - Are you using Redis and are the Redis env vars set in both the API and services pods? - What’s the response from the two curl calls above, especially the events filter? If you can share your Helm values or Deployments’ command/env sections, I can spot-check and suggest exact changes.
l
@Marvin I've noticed that in kubernetes it takes 10 to 15 seconds between the completion of flow submission and starting the process. How can i speed this up? e.g. logs:
Copy code
Worker 'KubernetesWorker dbaded29-ee8a-411d-8def-701155cf59b6' submitting flow run '36ecb0ec-1d50-46d5-af9c-5d91f161c29f'
03:47:14 PM
prefect.flow_runs.worker

Creating Kubernetes job...
03:47:15 PM
prefect.flow_runs.worker

Completed submission of flow run '36ecb0ec-1d50-46d5-af9c-5d91f161c29f'
03:47:15 PM
prefect.flow_runs.worker

Opening process...
03:47:27 PM
prefect.flow_runs.runner

 > Running set_working_directory step...
03:47:31 PM
prefect.flow_runs

Beginning flow run 'rampant-porcupine' for flow 'timocom-search-flow'
m
✅ thought for 664.6 seconds
You’re seeing the normal Kubernetes cold start gap. The worker submits the job immediately, but there’s a 10–15s window while K8s schedules the pod, pulls the image, starts the container, and Prefect initializes inside it. You can usually cut this down with a few changes: Where the time is spent - Scheduler placement: 1–3s (can be higher if nodes are busy or autoscaling) - Image pull: 2–10s (often the biggest factor on first run or large images) - Container start + Prefect runtime init: 1–3s - Optional runtime code/deps fetch: 2–10s if you’re cloning/installing at run time Practical ways to speed it up 1) Use a pre-built image with your code and dependencies - Build and push an image for your deployment; reference it in the work pool/deployment so the pod does not need to pip install or git clone at run time. - In Prefect 3.x:
Copy code
flow.from_source("<https://github.com/org/repo>", entrypoint="path/to/flow.py:my_flow").deploy(
  name="my-deployment",
  work_pool_name="kubernetes-pool",
  image="my-registry/my-flow:1.0.0"  # pre-built with code+deps
)
Docs: Deploy via Python, Store flow code 2) Ensure fast image pulls (or avoid pulls) - Set 
image_pull_policy
to 
IfNotPresent
in your Kubernetes work pool so cached images are reused. - Pre-pull images on nodes (DaemonSet) or keep nodes warm so the image is cached. - Use a registry in the same region/VPC (ECR for EKS, GAR/GCR for GKE, ACR for AKS). - Keep the image small and stable (pin tags) to maximize cache hits. 3) Give the scheduler something to place quickly - Add resource requests in your work pool job template so the pod schedules faster:
Copy code
spec:
  template:
    spec:
      containers:
      - name: flow
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"
- If you use cluster autoscaling, keep a small baseline of warm nodes (min nodes > 0) to avoid scale-up delays. 4) Don’t install anything at runtime - Avoid 
EXTRA_PIP_PACKAGES
or 
pip install
in init containers. Bake everything into the image. 5) If runs are scheduled, start earlier (prefetch) - This doesn’t reduce absolute startup time; it ensures the job is ready on time by submitting earlier:
Copy code
prefect worker start --pool <your-pool> --prefetch-seconds 30
# or
PREFECT_WORKER_PREFETCH_SECONDS=30
Docs: Workers overview 6) Kubernetes worker-specific knobs (for reliability, not raw speed) - `image_pull_policy`: set to 
IfNotPresent
- `pod_watch_timeout_seconds`: increase only if you use slow/serverless nodes; it’s a timeout, not a speed-up - If you’re creating a Secret for the API key on every run, pre-create it and reference it to avoid extra API calls Kubernetes worker docs - Kubernetes worker guide - prefect-kubernetes integration If you share: - Your work pool’s job template (image, image_pull_policy, resources) - Whether you install packages or fetch code at runtime - Cluster details (EKS/GKE/AKS, autoscaler settings, image registry location) …I can suggest exact edits to get you from ~10–15s down to ~3–5s in most clusters.
l
@Marvin adding resources in the job template made te startup even slower. The image pull policy is already IfNotPresent.
m
Your message was too long, here's your message at the allowed limit: adding resources in the job template made te startup even slower The image pull policy is already IfNotPresent and here is the base-job-template json ```{ job_configuration { command {{ command }} env {{ env }} labels {{ labels }} name {{ name }} namespace {{ namespace }} job_manifest { apiVersion batch/v1 kind Job metadata { generateName {{ name }}- labels {{ labels }} namespace {{ namespace }} } spec { backoffLimit {{ backoff_limit }} completions 1 parallelism 1 template { spec { restartPolicy Never imagePullSecrets [ { name harbor-creds } ] serviceAccountName {{ service_account_name }} containers [ { name prefect-job image {{ image }} imagePullPolicy {{ image_pull_policy }} resources { requests { cpu 100m memory 128Mi } limits { cpu 500m memory 512Mi } } args {{ command }} env {{ env }} } ] } } ttlSecondsAfterFinished {{ finished_job_ttl }} } } cluster_config {{ cluster_config }} job_watch_timeout_seconds {{ job_watch_timeout_seconds }} pod_watch_timeout_seconds {{ pod_watch_timeout_seconds }} stream_output {{ stream_output }} } variables { description Default variables for the Kubernetes worker \n\nThe schema for this class is used to populate the 
variables
section of the default base job template type object properties { name { anyOf [ { type string } { type null } ] default null description Name given to infrastructure created by a worker title Name } env { type object title Environment Variables description Environment variables to set when starting a flow run additionalProperties { anyOf [ { type string } { type null } ] } } labels { type object title Labels description Labels applied to infrastructure created by a worker additionalProperties { type string } } command { anyOf [ { type string } { type null } ] default null title Command description The command to use when starting a flow run Usually left blank } namespace { type string default prefect-server-eng title Namespace description The Kubernetes namespace to create jobs within } image { anyOf [ { type string } { type null } ] default null title Image description Container image for the job examples [ harbor mycompany local/prefect/flow latest ] } service_account_name { anyOf [ { type string } { type null } ] default null title Service Account Name description The Kubernetes service account to use for job creation } image_pull_policy { type string default IfNotPresent enum [ IfNotPresent Always Never ] title Image Pull Policy description The Kubernetes image pull policy to use for job containers } backoff_limit { type integer default 0 minimum 0 title Backoff Limit description How many times Kubernetes retries a job after pod eviction } finished_job_ttl { anyOf [ { type integer } { type null } ] default 3600 title Finished Job TTL description Seconds to retain jobs after completion } job_watch_timeout_seconds { anyOf [ { type integer } { type null } ] default null title Job Watch Timeout Seconds description Seconds to wait for each event emitted by a job before timing out } pod_watch_timeout_seconds { type integer default 60 title
✅ thought for 37.6 seconds
Got it — if adding “resources” made things slower, you likely increased pod requests enough that the scheduler had fewer placement options or had to scale up nodes. That adds seconds to minutes depending on your cluster. What typically causes the slowdown after adding resources - Requests vs limits behavior: if you set limits without explicit requests, Kubernetes defaults request = limit. A high limit can become a high request, making the pod harder to place. - Cluster capacity/fragmentation: larger requests need larger contiguous allocatable CPU/memory; scheduler may wait or trigger cluster-autoscaler. - Node image cache miss: with multiple node pools/nodes, your pod may land on a node that doesn’t have your image cached; 
IfNotPresent
still pulls if the image isn’t on that node. What to do instead - Set small explicit requests and higher limits (Burstable QoS) - Example for the flow container in your work pool job template:
Copy code
resources:
    requests:
      cpu: "50m"         # or 100m
      memory: "128Mi"
    limits:
      cpu: "500m"        # or what you really need
      memory: "512Mi"
This keeps scheduling flexible but still lets the pod burst. - Keep constraints minimal - Avoid adding strict nodeSelector/affinity/taints unless necessary — they reduce eligible nodes and can slow scheduling. - Pre-pull your image on nodes - Use a DaemonSet to cache your flow image on every node so the first run on each node does not pay the pull penalty.
Copy code
apiVersion: apps/v1
  kind: DaemonSet
  metadata:
    name: prefect-image-cache
  spec:
    selector:
      matchLabels: {app: prefect-image-cache}
    template:
      metadata: {labels: {app: prefect-image-cache}}
      spec:
        containers:
        - name: pull
          image: your-registry/your-image:tag
          command: ["sh","-c","sleep infinity"]
- Or keep a small number of nodes warm that already have the image cached. - Keep the image small and stable - Pre-build your deployment image with code+deps and reuse a pinned tag so layers stay cached. - If you use multiple pools/node groups, prefer landing pods on a pool where the image is already cached (via node labels/affinity as a gentle preference, not a hard constraint). - If the runs are scheduled, submit earlier to mask infra latency - Increase worker prefetch:
Copy code
prefect worker start --pool <your-pool> --prefetch-seconds 30
Or set 
PREFECT_WORKER_PREFETCH_SECONDS=30
. How to confirm what’s causing the delay - Describe the pod right after submission:
Copy code
kubectl describe pod <your-pod>
Look for: - “0/X nodes are available: Insufficient cpu/memory” → requests too high or cluster needs to scale - “Pulling image …” with long gap → cache miss or slow registry - Affinity/selector/taints mismatches - Check timestamps of events (Scheduled, Pulling, Pulled, Created, Started) to see which step is slowest. If you can share: - The resources section you added (requests/limits) - A 
kubectl describe pod
snippet showing the events for one slow-start run - Your cluster type (EKS/GKE/AKS) and whether autoscaling/serverless nodes are in use …I’ll tailor exact job template changes (and, if needed, a DaemonSet + mild affinity example) to get you back under ~5 seconds.
l
@Marvin Does @flow(task_runner=ThreadPoolTaskRunner()) behave the same way in a process worker as in kubernetes worker?
m
✅ thought for 158.8 seconds
Short answer: yes. In Prefect 3.x the task runner is a flow-level concern and behaves the same regardless of worker type. - 
@flow(task_runner=ThreadPoolTaskRunner(...))
controls how tasks within that single flow run execute (thread-based concurrency inside the flow’s Python process). - A process worker runs your flow in a subprocess; a Kubernetes worker runs it in a pod. In both cases, the flow code (including the task runner) runs inside that environment and uses the same thread pool semantics. - Differences you may see in timing are due to infrastructure startup (pod scheduling, image pull, node warmup), not the task runner itself. Docs: Task runners
16 Views
Open in Slack
PreviousNext
Powered by