https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> In Prefect 2.20, when using kubernete...
# marvin-ai
f
@Marvin In Prefect 2.20, when using kubernetes work pool and workers, how can I reduce the time it takes for a run to go from pending to running state?
m
✅ thought for 126.9 seconds
In Prefect 2.20 with a Kubernetes work pool + worker, the delay from 
Pending
to 
Running
usually comes from two separate things: 1. Worker polling latency — how quickly a worker notices and claims the run 2. Kubernetes job/pod startup latency — how long the pod takes to get scheduled, pull the image, and start For Kubernetes workers, pod startup is often the biggest factor. What you can tune 1. Reduce worker polling interval Workers poll Prefect periodically for scheduled work. Lowering that interval can reduce the wait before a run is picked up. Set:
Copy code
bash
export PREFECT_WORKER_QUERY_SECONDS=5
Default is typically 
10
, so cutting it to 
5
can shave off several seconds. You can also tune:
Copy code
bash
export PREFECT_WORKER_PREFETCH_SECONDS=10
or a bit higher if useful for your scheduling pattern. 2. Make Kubernetes pods start faster This is usually the highest-impact optimization. Things to check: - Use a smaller image - Make sure the image is already cached on nodes - Set 
imagePullPolicy: IfNotPresent
- Avoid expensive container startup logic - Keep resource requests modest so pods schedule faster - Avoid unnecessary init containers / sidecars if possible If every flow run launches a new Kubernetes job, image pull and scheduling time can dominate. 3. Keep workers available and unconstrained Make sure the worker itself is not bottlenecked: - Run enough worker replicas - Increase the worker 
--limit
if it is too low - Check work pool / work queue concurrency limits - Make sure the work pool and queue are not paused If concurrency limits are saturated, runs can sit waiting even after being discovered. 4. Check pool / queue concurrency limits If your work pool or queue has a concurrency limit, new runs may remain queued until a slot opens. This can look like “pending is slow” when the real issue is no available execution slot. 5. Separate latency-sensitive work If you have short, urgent flows, put them in a dedicated work pool / queue so they are not competing with longer batch jobs. Practical recommendations If your goal is faster handoff, I’d start with:
Copy code
bash
export PREFECT_WORKER_QUERY_SECONDS=5
export PREFECT_WORKER_PREFETCH_SECONDS=10
and then focus on Kubernetes startup: - small image - cached image - 
IfNotPresent
- lower scheduling overhead Rule of thumb - If the run spends time before the worker claims it: tune worker polling - If the worker claims it quickly but it still takes a while to become `Running`: tune Kubernetes pod/job startup If you want, I can also give you a 2.20-specific checklist for Kubernetes work pools, or help you inspect whether your delay is mostly Prefect polling vs Kubernetes startup.
f
@Marvin Is there any improvement expected in this behavior with Prefect 3?
m
✅ thought for 191.6 seconds
Short answer: not much for this specific behavior. In Prefect 3, there are architectural improvements to workers, but for a Kubernetes work pool / worker setup, you should not expect a major built-in reduction in the 
Pending
→ 
Running
delay just from upgrading. What improved in Prefect 3 Prefect 3 workers gained better worker/server coordination, including newer worker-channel improvements in recent 3.x versions. That helps with things like: - worker heartbeats - work pool config synchronization - cancellation delivery - HA/race-condition behavior But the important caveat is: scheduled flow-run pickup is still polling-based So the part that often matters most for 
Pending
→ 
Running
remains broadly the same: - worker polls for work - worker submits Kubernetes infrastructure - Kubernetes schedules and starts the pod/job What did not materially improve For your exact question, these core costs are still present in 3.x: 1. Worker polling interval Workers still query for scheduled runs on an interval, controlled by settings like 
PREFECT_WORKER_QUERY_SECONDS
. 2. Kubernetes startup time Job creation, pod scheduling, image pull, and container startup are still major contributors. So if your current delay is mostly: - worker noticing the run late, or - Kubernetes taking time to start the pod, then Prefect 3 alone will not eliminate that. What to expect in practice If you're on Prefect 2.20 and move to Prefect 3: - Operational behavior may be better - Worker internals are more modern - Some edge cases may be smoother But for latency from pending to running, expect only minor or no noticeable improvement unless you also tune the system. Best ways to improve it in either 2.20 or 3.x Still the same levers: - lower 
PREFECT_WORKER_QUERY_SECONDS
- ensure enough workers / worker concurrency - check work pool and queue concurrency limits - optimize Kubernetes pod startup - use small images - cache images on nodes - use 
imagePullPolicy: IfNotPresent
- reduce scheduling overhead Bottom line Upgrading to Prefect 3 is not, by itself, likely to significantly reduce 
Pending
→ 
Running
time for Kubernetes workers. If you want, I can help you break this down into: 1. Prefect-side latency vs 2. Kubernetes-side latency and give you a quick way to measure which one is dominating in your setup.
f
@Marvin in Prefect 2.20, I would like to manage all my deployments, work queues, flows etc. with a simple configuration file such that all of them are tightly coupled and can be extended as new flows and deployments are added. Is there any such functionality in prefect 2.20?
m
✅ thought for 199.2 seconds
Yes — partially. In Prefect 2.20, the main feature for this is 
prefect.yaml
, which lets you manage deployments declaratively in a single file and keep them tightly coupled as your project grows. What 
prefect.yaml
is good for: - defining multiple deployments - setting each deployment’s: - 
entrypoint
- parameters - schedules - tags - work pool reference - work queue name - job variables - defining shared build / push / pull steps - using templating, YAML anchors, variables, env vars, and block references for reuse So if your goal is: - “I want one config file for many flows/deployments” - “I want shared defaults and extensibility” - “I want new deployments to be easy to add” then yes, 
prefect.yaml
is the closest built-in solution in Prefect 2.20. A rough example:
Copy code
yaml
prefect-version: "2.20.0"
name: my-project

definitions:
  work_pool: &k8s_pool
    name: my-k8s-pool
    work_queue_name: default
    job_variables:
      image: my-image:latest

deployments:
  - name: flow-a-dev
    entrypoint: flows/flow_a.py:flow_a
    tags: ["dev"]
    work_pool: *k8s_pool

  - name: flow-b-prod
    entrypoint: flows/flow_b.py:flow_b
    tags: ["prod"]
    schedule:
      cron: "0 * * * *"
    work_pool: *k8s_pool
But there’s an important limitation: 
prefect.yaml
does not fully manage everything in your Prefect workspace. In particular, in 2.20 it does not declaratively create/manage: - work pools themselves - advanced work queue lifecycle/config - blocks - variables - all other workspace objects It can reference work pools and queues, but those generally need to exist already or be managed separately. So the practical answer is: Prefect 2.20 has deployment-as-config via 
prefect.yaml
, but not full “everything-as-code” for all Prefect objects. If you want a tightly coupled setup in 2.20, the usual pattern is: 1. Use 
prefect.yaml
 as the source of truth for deployments 2. Create/bootstrap work pools / queues / blocks / variables separately via: - CLI - Python scripts using the Prefect API/client - Terraform / Kubernetes / CI bootstrap scripts, if applicable A common approach is: - 
prefect.yaml
for deployments - one 
bootstrap.py
or shell script for workspace resources If you want, I can show you a good project structure for Prefect 2.20 that combines: - 
prefect.yaml
- multiple flows - Kubernetes work pool - reusable deployment definitions - a bootstrap script for pools/queues/blocks Docs: - Prefect 2.20 deployment guide - Prefect 2.20 deployments
f
@Marvin yes please show a good project structure for Prefect 2.20 that combines: - 
prefect.yaml
- multiple flows - Kubernetes work pool - reusable deployment definitions - a bootstrap script for pools/queues/blocks • Scenario where single worker is picking up from multiple queues, all linked to the same flow
m
✅ thought for 28.5 seconds
Absolutely — here’s a practical Prefect 2.20 project structure for your scenario: - multiple flows - 
prefect.yaml
- Kubernetes work pool - reusable deployment definitions - bootstrap script for pools / queues / blocks - one worker polling multiple queues - multiple deployments all pointing to the same underlying flow --- Recommended project layout
Copy code
text
my-prefect-project/
├── flows/
│   ├── shared_pipeline.py
│   ├── ingest.py
│   └── cleanup.py
├── deployment/
│   ├── prefect.yaml
│   ├── bootstrap.py
│   └── settings.yaml
├── Dockerfile
├── requirements.txt
└── README.md
--- What each piece does - 
flows/
- contains your actual flow code - can have one reusable flow used by many deployments - 
deployment/prefect.yaml
- defines deployments declaratively - 
deployment/bootstrap.py
- creates work pool, work queues, and blocks - 
deployment/settings.yaml
- your own higher-level config for queues/environments - 
Dockerfile
- image used by Kubernetes jobs --- Example flow: one flow, many deployments, many queues Suppose you want a single flow that behaves differently depending on deployment/queue/environment. 
flows/shared_pipeline.py
Copy code
python
from prefect import flow, get_run_logger


@flow(name="shared-pipeline")
def shared_pipeline(dataset: str = "default", target: str = "dev"):
    logger = get_run_logger()
    <http://logger.info|logger.info>(f"Running shared pipeline for dataset={dataset}, target={target}")

    # your actual logic here
    return {
        "dataset": dataset,
        "target": target,
        "status": "ok",
    }
You can then create many deployments for this same flow: - 
shared-pipeline-dev
- 
shared-pipeline-prod
- 
shared-pipeline-priority
- etc. Each deployment can point to: - a different queue - different parameters - different tags - different schedules --- Project-level config for bootstrap 
deployment/settings.yaml
Copy code
yaml
work_pool:
  name: k8s-shared-pool
  type: kubernetes

queues:
  - name: default
    priority: 1
  - name: high-priority
    priority: 2
  - name: backfill
    priority: 3

blocks:
  kubernetes_cluster_config:
    name: in-cluster-config

worker:
  name: shared-worker
  queues:
    - default
    - high-priority
    - backfill
This file is your own config, not a native Prefect file, but it’s useful for keeping bootstrap logic centralized. --- Bootstrap script for pools / queues / blocks In Prefect 2.20, work pools/queues/blocks are best bootstrapped separately. 
deployment/bootstrap.py
```python import asyncio import yaml from prefect.client.orchestration import get_client from prefect.client.schemas.actions import WorkPoolCreate from prefect.blocks.kubernetes import KubernetesClusterConfig SETTINGS_FILE = "deployment/settings.yaml" async def ensure_work_pool(client, name: str, pool_type: str): existing = await client.read_work_pool(name) if existing: print(f"Work pool already exists: {name}") return existing pool = await client.create_work_pool( work_pool=WorkPoolCreate( name=name, type=pool_type, description="Shared Kubernetes pool for multiple queues and deployments", ) ) print(f"Created work pool: {name}") return pool async def ensure_work_queue(client, pool_name: str, queue_name: str, priority: int): try: queue = await client.read_work_queue_by_name( work_pool_name=pool_name, name=queue_name, ) print(f"Work queue already exists: {queue_name}") return queue except Exception: queue = await client.create_work_queue( name=queue_name, work_pool_name=pool_name, priority=priority, ) print(f"Created work queue: {queue_name}") return queue async def ensure_kubernetes_block(name: str): try: await KubernetesClusterConfig.load(name)
print(f"Kubernetes block already exists: {name}") except Exception: block = KubernetesClusterConfig( config={}, # in-cluster config example context_name=None, ) await block.save(name, overwrite=True) print(f"Created Kubernetes block: {name}") async def main(): with open(SETTINGS_FILE, "r") as f: settings = yaml.safe_load(f) pool_name = settings["work_pool"]["name"] pool_type = settings["work_pool"]["type"] await ensure_kubernetes_block( settings["blocks"]["kubernetes_cluster_config"]["name"] ) async with get_client() as client: await ensure_work_pool(client, pool_name, pool_type) for queue in settings["queues"]: await ensure_work_queue( client, pool_name, queue["name"], queue["priority"], ) if name == "__main__": asyncio.run(main())
Copy code
*Note:* depending on your exact 2.20 patch version, some client methods may differ slightly. If you want, I can give you a version that uses only CLI commands instead of the Python client.

---

*Reusable `prefect.yaml`*

This is where you define deployments for multiple flows and multiple queues.

`deployment/prefect.yaml`

```yaml
prefect-version: "2.20.0"
name: "my-prefect-project"

definitions:
  tags:
    common: &common_tags
      - kubernetes
      - shared-worker

  pull_steps:
    common: &common_pull
      - prefect.deployments.steps.git_clone:
          repository: "<https://github.com/my-org/my-prefect-project.git>"
          branch: "main"
      - prefect.deployments.steps.set_working_directory:
          directory: "/opt/prefect/my-prefect-project"

  k8s_pool_default: &k8s_pool_default
    name: "k8s-shared-pool"
    work_queue_name: "default"
    job_variables:
      image: "my-registry/my-prefect-project:latest"
      namespace: "prefect"
      image_pull_policy: "IfNotPresent"

  k8s_pool_high_priority: &k8s_pool_high_priority
    name: "k8s-shared-pool"
    work_queue_name: "high-priority"
    job_variables:
      image: "my-registry/my-prefect-project:latest"
      namespace: "prefect"
      image_pull_policy: "IfNotPresent"

  k8s_pool_backfill: &k8s_pool_backfill
    name: "k8s-shared-pool"
    work_queue_name: "backfill"
    job_variables:
      image: "my-registry/my-prefect-project:latest"
      namespace: "prefect"
      image_pull_policy: "IfNotPresent"

deployments:
  - name: "shared-pipeline-dev"
    entrypoint: "flows/shared_pipeline.py:shared_pipeline"
    parameters:
      dataset: "customers"
      target: "dev"
    tags: *common_tags
    work_pool: *k8s_pool_default
    pull: *common_pull

  - name: "shared-pipeline-prod"
    entrypoint: "flows/shared_pipeline.py:shared_pipeline"
    parameters:
      dataset: "customers"
      target: "prod"
    tags: *common_tags
    schedule:
      cron: "0 * * * *"
      timezone: "UTC"
    work_pool: *k8s_pool_high_priority
    pull: *common_pull

  - name: "shared-pipeline-backfill"
    entrypoint: "flows/shared_pipeline.py:shared_pipeline"
    parameters:
      dataset: "customers_historical"
      target: "prod"
    tags:
      - kubernetes
      - backfill
    work_pool: *k8s_pool_backfill
    pull: *common_pull

  - name: "cleanup-daily"
    entrypoint: "flows/cleanup.py:cleanup_flow"
    tags: *common_tags
    schedule:
      cron: "0 3 * * *"
      timezone: "UTC"
    work_pool: *k8s_pool_default
    pull: *common_pull
This gives you: - shared job config via YAML anchors - same work pool, different queues - same flow, multiple deployments - clean extensibility when adding more deployments --- Single worker polling multiple queues Your worker can poll multiple queues in the same work pool. Example: ```bash prefect worker start \ --pool k8s-shared-pool \ --work-queue default \ --work-queue high-priority \ --work-queue backfill \ --name shared-worker \
--type kubernetes
Copy code
This means:
- one worker process
- one Kubernetes work pool
- multiple queues
- deployments routed by queue name

If your CLI variant prefers a comma-separated value or repeated queue args can differ slightly by version, check:

```bash
prefect worker start --help
In general, the idea is: one worker can watch several queues from the same pool. --- How this supports “same flow across multiple queues” This is a very good pattern when: - the same code path serves different workloads - you want routing and prioritization without duplicating flow logic For example: - 
shared-pipeline-prod
→ 
high-priority
- 
shared-pipeline-dev
→ 
default
- 
shared-pipeline-backfill
→ 
backfill
All three deployments use:
Copy code
text
flows/shared_pipeline.py:shared_pipeline
but differ in: - parameters - schedule - queue - tags That’s usually cleaner than making separate flow functions. --- Recommended operational model I’d suggest this workflow: 1. Bootstrap workspace resources Run:
Copy code
bash
python deployment/bootstrap.py
This ensures: - Kubernetes block exists - work pool exists - work queues exist 2. Register deployments Run:
Copy code
bash
prefect deploy --prefect-file deployment/prefect.yaml
3. Start worker Run:
Copy code
bash
prefect worker start --pool k8s-shared-pool --work-queue default --work-queue high-priority --work-queue backfill --type kubernetes
4. Add new deployments by editing only 
prefect.yaml
 Usually for a new deployment, you only add another block under 
deployments:
. --- When to use multiple queues vs multiple pools Use multiple queues in one pool when: - infrastructure type is the same - worker type is the same - you just want prioritization / logical separation Use multiple pools when: - infrastructure differs significantly - different base job templates are needed - different clusters/namespaces/security models are needed For your scenario, same pool + multiple queues is a good fit. --- Suggested 
cleanup.py
example
Copy code
python
from prefect import flow, get_run_logger


@flow(name="cleanup-flow")
def cleanup_flow():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Running cleanup")
--- Suggested Dockerfile
Copy code
dockerfile
FROM python:3.11-slim

WORKDIR /opt/prefect/my-prefect-project

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

ENV PYTHONPATH=/opt/prefect/my-prefect-project

CMD ["prefect", "worker", "start", "--pool", "k8s-shared-pool", "--work-queue", "default", "--work-queue", "high-priority", "--work-queue", "backfill", "--type", "kubernetes"]
--- Requirements example
Copy code
text
prefect==2.20.0
pyyaml
kubernetes
You may also need the relevant Prefect collection depending on your setup. --- Best practices for this pattern - Keep flow code reusable and let deployments specialize behavior - Use one 
prefect.yaml
 as the deployment source of truth - Use a bootstrap script for non-deployment resources - Use queues to separate priority/classes of work - Keep job variables anchored so image/namespace settings are reused - Add deployments, not duplicate flows, when behavior differences are mostly config-level --- One important caveat In Prefect 2.20, 
prefect.yaml
is best thought of as: - deployment-as-code not - full workspace-as-code So your design should intentionally split: - 
prefect.yaml
→ deployments - 
bootstrap.py
→ workspace resources That’s the cleanest 2.20 pattern. If you want, I can next give you either: 1. a fully corrected version using only CLI bootstrap commands or 2. a more production-ready Kubernetes example with: - namespace - service account - image - env vars - resource requests/limits - secrets/block references
Open in Slack
PreviousNext
Powered by