https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> i use currently prefect on a kubernet...
# ask-marvin
o
@Marvin i use currently prefect on a kubernetes clustr, i have non-ephemeral apps that register via serve to prefect, the non-ephemeral apps are deployed via helm charts and have their own service, pod etc, i want to know how i scale the non-ephemeral apps, through the cuncurrency settings in the prefet ui or through the helm chart? how do i make sure that prefect makes shure horizontal scaling is done properly? what does the communtiy do with these cases? is there a community persopective on it?
m
✅ thought for 199.6 seconds
Short version: scale the pods with Kubernetes/Helm; control run concurrency with Prefect. They solve different problems: - Helm / Kubernetes replicas / HPA → how many copies of your long-running app are running - Prefect concurrency limits → how many flow runs Prefect should allow at once So for non-ephemeral 
serve
-based apps, the usual pattern is: 1. Deploy your app via Helm as a normal long-running Deployment 2. Scale replicas with Kubernetes (
replicas
or HPA) 3. Use Prefect concurrency settings to cap how much work can run at once Prefect does not do Kubernetes horizontal autoscaling for 
serve
apps. Kubernetes does that. --- What Prefect concurrency does vs what Helm scaling does Prefect UI / deployment concurrency - Limits concurrent flow runs - Protects databases, APIs, downstream services - Coordinates execution at the orchestration layer - Good for “only 10 runs total at once” Helm / Kubernetes scaling - Adds/removes pods - Gives you more polling/execution capacity - Handles availability and resource scaling - Good for “run more copies of this service when CPU/memory rises” So if you want more capacity, increase pod replicas. If you want more allowed simultaneous work, increase Prefect concurrency. Often you want both. --- Important caveat for 
serve
and multiple replicas If you run multiple replicas of the same 
serve
process, all replicas may poll Prefect for work. Prefect’s orchestration is what prevents duplicate execution by coordinating state transitions and concurrency centrally. That means: - Yes, horizontal scaling can work - But you should think of Prefect as coordinating who gets the run, not as driving the Kubernetes scaling event itself In practice, to make this sane: - set a deployment/global concurrency limit in Prefect - optionally set a per-process limit in the runner/serve config - scale the Deployment with Kubernetes based on CPU/memory or custom metrics --- Recommended mental model Use: - Kubernetes/Helm for: “How many app instances exist?” - Prefect for: “How many runs may execute concurrently?” - Your app/resources for: “How much parallelism can one pod safely handle?” A simple formula: - 
total possible throughput ~= replicas × per-pod concurrency
- but actual execution should also be bounded by Prefect deployment concurrency if you set one So for example: - 4 replicas in Kubernetes - each pod can safely process 2 flow runs - Prefect deployment concurrency = 6 Then max actual concurrent runs should be 6, even though theoretical pod capacity is 8. --- What I would usually recommend For non-ephemeral apps on Kubernetes: Option A: one flow run at a time per pod - scale pod count with HPA - keep per-pod concurrency low - use Prefect deployment concurrency as the global cap This is the safest option if runs are heavy or not perfectly isolated. Option B: multiple flow runs per pod - only if your app is clearly designed for it - set a per-pod/process limit - still keep a Prefect global/deployment concurrency limit This is more efficient but easier to overload. --- Community perspective / common pattern The most common community pattern is generally: - use workers + work pools when you want Prefect to launch ephemeral infrastructure - use serve when you already have a long-lived service/process and just want it to poll for work For your setup, since you already have long-running Helm-managed apps, the community perspective is usually: - let Kubernetes own horizontal scaling - let Prefect own orchestration and concurrency - don’t expect Prefect UI concurrency settings to create or remove pods So if the question is “should I scale through Prefect UI or Helm chart?” the answer is: - Helm/Kubernetes for horizontal scaling - Prefect for concurrency control --- How to make sure horizontal scaling is “done properly” A good setup usually includes: 1. Deployment concurrency limit in Prefect
- prevents too many runs globally 2. Reasonable per-pod concurrency - avoid one pod grabbing too much work 3. Kubernetes HPA - scale on CPU/memory, or custom metrics if you have them 4. Idempotent flow/task design - always a good idea in distributed systems 5. Health/readiness probes - so only healthy pods poll and execute --- Rule of thumb If your concern is: - “too many flow runs hit my system” → Prefect concurrency - “one pod is overloaded” → reduce per-pod concurrency or add replicas - “I need more app instances” → Helm/HPA - “I want Prefect to create new compute automatically” → consider workers/work pools instead of 
serve
--- One subtle point If you want true infrastructure autoscaling driven by queued Prefect work, 
serve
is usually not the strongest model. That’s where work pools/workers and ephemeral execution are often a better fit, because Prefect is then much closer to the infrastructure lifecycle. With 
serve
, your app is already running, so Kubernetes is the natural scaler. --- I didn’t find a specific GitHub discussion that neatly codifies a single “official community answer” for this exact pattern, but the practical guidance is strongly in this direction: - Kubernetes scales the service - Prefect limits/coordinates the work If you want, I can help you design a concrete setup for your case, for example: - 1 deployment, N replicas, 1 run per pod - HPA + Prefect concurrency - or whether you should switch from 
serve
to workers/work pools for better scaling behavior
o
how do i do ?Option A: one flow run at a time per pod - scale pod count with HPA - keep per-pod concurrency low - use Prefect deployment concurrency as the global cap This is the safest option if runs are heavy or not perfectly isolated.
@Marvin how do i how do i do in best practie?Option A: one flow run at a time per pod - scale pod count with HPA - keep per-pod concurrency low - use Prefect deployment concurrency as the global cap This is the safest option if runs are heavy or not perfectly isolated.
m
✅ thought for 192.2 seconds
Yes — if you want best practice for heavy, non-isolated 
serve
apps on Kubernetes, I’d implement it like this: Target behavior - each pod handles at most 1 flow run at a time - Kubernetes HPA scales the number of pods - Prefect enforces the global max concurrent runs - pods expose health checks so Kubernetes only keeps healthy pollers --- Best-practice architecture Think in 3 layers: 1. Per pod capacity → set Prefect 
limit=1
2. Global system capacity → set Prefect 
global_limit=N
3. Horizontal scaling → set Helm/Kubernetes 
replicas
+ HPA So: - 
limit=1
means one pod won’t run two flow runs concurrently - 
global_limit=10
means all pods together won’t exceed 10 concurrent runs - HPA can scale from e.g. 1 → 10 replicas That gives you a very predictable model: - 1 pod = 1 active run max - 5 pods = 5 active runs max - if HPA scales to 12 pods but 
global_limit=10
, only 10 runs should execute at once --- Recommended Prefect code Use 
flow.serve(...)
like this:
Copy code
python
from prefect import flow

@flow(log_prints=True)
def my_flow(customer_id: str):
    print(f"Processing {customer_id}")
    # your work here

if __name__ == "__main__":
    my_flow.serve(
        name="my-flow",
        limit=1,
        global_limit=10,
        pause_on_shutdown=False,
        webserver=True,
        cron="*/5 * * * *",
    )
Why these settings? - 
limit=1
→ one run at a time per pod - 
global_limit=10
→ max 10 concurrent runs across all replicas - 
pause_on_shutdown=False
→ better for Kubernetes restarts/rollouts - 
webserver=True
→ gives you 
/health
for readiness/liveness probes --- If you want the more explicit global concurrency config You can also use the richer config form:
Copy code
python
from prefect import flow
from prefect.client.schemas.objects import ConcurrencyLimitConfig, ConcurrencyLimitStrategy

@flow
def my_flow():
    pass

if __name__ == "__main__":
    my_flow.serve(
        name="my-flow",
        limit=1,
        global_limit=ConcurrencyLimitConfig(
            limit=10,
            collision_strategy=ConcurrencyLimitStrategy.ENQUEUE,
            grace_period_seconds=300,
        ),
        pause_on_shutdown=False,
        webserver=True,
    )
That’s useful if you want queued runs to wait instead of being canceled when the concurrency limit is full. --- Best-practice Kubernetes deployment Your Helm chart / Deployment should manage replicas and health checks. A good baseline Deployment looks like:
Copy code
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-flow
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-flow
  template:
    metadata:
      labels:
        app: my-flow
    spec:
      containers:
      - name: my-flow
        image: my-registry/my-flow:latest
        env:
        - name: PREFECT_API_URL
          value: "<https://api.prefect.cloud/api>"
        - name: PREFECT_API_KEY
          valueFrom:
            secretKeyRef:
              name: prefect-api-key
              key: token
        - name: PREFECT_RUNNER_PROCESS_LIMIT
          value: "1"
        ports:
        - name: http
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 30
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "1"
            memory: "2Gi"
A few notes: - 
PREFECT_RUNNER_PROCESS_LIMIT=1
is equivalent to `limit=1`; I’d usually set it in code or env, not both unless you want env-driven overrides - readiness/liveness probes matter a lot for long-lived pollers - set resource requests realistically, because HPA depends on them --- Best-practice HPA Then let Kubernetes scale the number of pods.
Example:
Copy code
yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-flow
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-flow
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
Best practice here: - start with CPU/memory-based HPA - don’t overcomplicate with custom metrics first - tune based on real flow runtime behavior --- How to choose the numbers A simple approach: Step 1: determine safe per-pod concurrency - if flows are heavy or stateful → use 
1
- that’s your case, so 
limit=1
Step 2: determine safe global concurrency Ask: - how many runs can your downstream systems handle? - how many DB connections / API calls / memory-heavy jobs can the cluster safely support? Example: - each run needs ~1.5 CPU and ~2 GB RAM - cluster safely supports 8 of them - downstream DB safely supports 6 - then set 
global_limit=6
Use the smallest real bottleneck, not the optimistic one. Step 3: set HPA bounds - `minReplicas`: enough to avoid cold starts / increase availability - `maxReplicas`: enough to match 
global_limit
or slightly exceed it If 
limit=1
, a clean rule is: - 
maxReplicas ~= global_limit
Example: - 
limit=1
- 
global_limit=8
- 
minReplicas=2
- 
maxReplicas=8
or 
10
--- My recommended default pattern For heavy flows: - 
limit=1
- 
global_limit=number your systems can really tolerate
- 
minReplicas=1 or 2
- 
maxReplicas=global_limit
- CPU/memory requests set accurately - HPA on CPU + memory - health probes enabled - flows designed to be idempotent if possible --- What happens operationally Example: - 
limit=1
- 
global_limit=5
- HPA can scale to 5 pods Then: - Pod A can run 1 flow - Pod B can run 1 flow - ... - total max = 5 If a 6th run is scheduled: - Prefect will not let it run immediately - it should wait for a concurrency slot, depending on your concurrency strategy If HPA scales up faster than work arrives: - extra pods just poll idle - that’s fine, though maybe slightly wasteful If HPA is too slow: - scheduled runs may queue in Prefect waiting for pods/slots --- Important best-practice caveat 
serve
is okay for this pattern, but it is still a long-lived poller model. That means: - Kubernetes scales the app - Prefect coordinates work - but Prefect does not directly ask Kubernetes for “one new pod per run” So if your real goal is: - “I want one Kubernetes pod per flow run” - “I want queued Prefect work to naturally map to new compute” - “I want stronger isolation between runs” then workers + Kubernetes work pools are usually the cleaner architecture. For your current setup though, if you want to stay with 
serve
, the 
limit=1
+ 
global_limit=N
+ HPA pattern is the safest way. --- What I would do in your shoes If you want to keep your current non-ephemeral Helm-deployed services: 1. Keep the app as a Deployment 2. Add 
webserver=True
3. Set 
limit=1
4. Set 
global_limit
in the deployment 5. Add readiness/liveness probes on 
/health
6. Add HPA on CPU/memory 7. Set 
maxReplicas
roughly equal to your desired global concurrency 8. Load test and tune from there --- Concrete example recommendation If you’re unsure where to start, start with this: - 
limit=1
- 
global_limit=3
- 
minReplicas=2
- 
maxReplicas=3
- HPA target CPU 70% - HPA target memory 80% Then observe: - are runs queueing too much? increase 
global_limit
- are pods underutilized? lower 
minReplicas
- are pods OOMing or thrashing? increase memory or reduce workload size --- Community-style recommendation in one sentence For long-running 
serve
apps on Kubernetes, the safest community-style pattern is:
one run per pod, Kubernetes scales replicas, Prefect caps total concurrency. --- Useful reference areas from the Prefect API surface: - 
prefect.flow.serve
- 
prefect global_concurrency_limit
- 
prefect concurrency_limit
If you want, I can next give you one of these: 1. a complete Helm values.yaml example for this pattern 2. a full production-ready Python 
serve
script 3. a decision guide: stay on 
serve
vs move to Kubernetes workers/work pools
o
@Marvin and how we make sure each app starts immediatly is that done with ?
Copy code
- name: PREFECT_RUNNER_PROCESS_LIMIT
          value: "1"
or with the deployment options
Copy code
name="my-flow",
        limit=1,
        global_limit=10,
        pause_on_shutdown=False,
        webserver=True,
        cron="*/5 * * * *",
also i do not get why a cron is inserted here?
m
✅ thought for 23.8 seconds
PREFECT_RUNNER_PROCESS_LIMIT=1
and 
limit=1
do not make apps start immediately. They only mean: - this pod may run at most 1 flow run at a time So they control concurrency inside the pod, not startup speed. --- What controls “starts immediately”? That depends on what you mean: If you mean “the app pod should already be running and ready” - that is controlled by Kubernetes replicas / HPA / startup time - keep at least 
minReplicas: 1
or more - use readiness probes - keep the app warm If you mean “a scheduled/manual flow run should begin execution immediately when submitted” - you need: - at least one ready pod already running - available Prefect concurrency slot - no backlog of older runs - polling interval not too slow So immediate start is mostly affected by: - pod availability - pod readiness - Prefect polling cadence - concurrency availability --- About these two settings These are basically two ways to set the same per-runner limit: In environment variables
Copy code
yaml
- name: PREFECT_RUNNER_PROCESS_LIMIT
  value: "1"
In Python
Copy code
python
my_flow.serve(
    name="my-flow",
    limit=1,
    ...
)
I would usually recommend: - set it in code if you want the behavior tied to the app definition - set it in env/Helm values if you want ops to tune it without rebuilding code But conceptually they are about max concurrent flow runs per pod, not immediate startup. --- Why I would not use both unless you want overrides If you define both, it can become confusing for your team: - “is concurrency controlled in code?” - “or in Helm values?” - “which one are we changing in production?” Best practice: - choose one source of truth for per-pod concurrency For example: - application-owned: use 
limit=1
in code - platform-owned: use 
PREFECT_RUNNER_PROCESS_LIMIT=1
in Helm --- Why was 
cron
in the example? Only because I showed a scheduled deployment example. You do not need 
cron
 unless you want Prefect to schedule runs automatically. 
cron="*/5 * * * *"
means: - Prefect should create a run every 5 minutes If your flows are started another way, for example: - manually from UI/API - via automations - event triggers - external systems calling Prefect then remove the cron. So your actual 
serve
may look more like:
Copy code
python
from prefect import flow

@flow
def my_flow(customer_id: str):
    pass

if __name__ == "__main__":
    my_flow.serve(
        name="my-flow",
        limit=1,
        global_limit=10,
        pause_on_shutdown=False,
        webserver=True,
    )
That registers the deployment and keeps the runner alive, without creating scheduled runs on a cron. --- So how do you get “immediate” starts in practice? For 
serve
, the practical recipe is: 1. Keep at least one pod always running - 
minReplicas: 1
or 
2
2. Make pods become ready quickly - lightweight image - fast startup - good readiness probes 3. Keep polling interval reasonable - don’t make polling too infrequent 4. Keep 
limit=1
 - so one pod handles one run at a time 5. Use HPA for scale-out - extra pods help absorb load, but they do not eliminate the need for one warm pod --- Important subtlety If you need true immediate scale-from-zero per run, 
serve
is not ideal. Because with `serve`: - the app must already be running to poll and pick up work So if there are zero pods, no run starts immediately. For that kind of model, workers + Kubernetes work pools are usually better, because Prefect can submit infrastructure for runs instead of waiting for a long-running app to poll. --- Best-practice answer to your exact question - 
PREFECT_RUNNER_PROCESS_LIMIT=1
→ not for immediate start, only per-pod concurrency - 
limit=1
→ same purpose, also not for immediate start - 
global_limit=10
→ global concurrency cap, also not for immediate start - 
cron=...
→ only if you want scheduled runs; otherwise remove it
If your goal is fast pickup of runs, focus on: - 
minReplicas > 0
- fast pod startup - readiness probes - a warm app already running --- If you want, I can give you: - a minimal 
serve
config without cron - or a Helm Deployment + HPA example optimized for fast pickup
o
@Marvin what i do not get, if each of the horizontal pods sets These are basically two ways to set the same per-runner limit: In environment variables
Copy code
yaml
- name: PREFECT_RUNNER_PROCESS_LIMIT
  value: "1"
In Python
Copy code
python
my_flow.serve(
    name="my-flow",
    limit=1,
    ...
)
@Marvin with a value like limit 1, is then each horizontal pod truly limited to one? and how does the server know that there are different serve pods to the same deployment name?
m
✅ thought for 200.9 seconds
Yes — with an important distinction: *`limit=1` or 
PREFECT_RUNNER_PROCESS_LIMIT=1
means one concurrent run per serve process / per pod, not one for the whole deployment across all pods.* So if you have 3 replicas, and each pod is running the same 
serve(...)
with 
limit=1
, then you can get up to 3 concurrent flow runs total. Example:
Copy code
python
my_flow.serve(
    name="my-flow",
    limit=1,
)
or
Copy code
yaml
- name: PREFECT_RUNNER_PROCESS_LIMIT
  value: "1"
If that same setup is running in 4 pods, then the effective capacity is:
Copy code
text
1 run per pod × 4 pods = up to 4 concurrent runs
How Prefect handles multiple serve pods for the same deployment The server does not really think in terms of “this is pod A” and “this is pod B” as durable registered serve instances. Instead, each serve process independently: 1. polls for scheduled runs for that deployment 2. sees runs that are ready 3. tries to claim one by proposing a state change 4. the server accepts that claim for only one runner So multiple pods may all see the same scheduled run, but only one of them successfully transitions it into the runnable state. The others lose the race and move on. So the coordination is basically:
Copy code
text
many serve pods poll
-> all may notice the same scheduled run
-> only one can successfully claim it
-> that pod executes it
Answering your two specific questions 1. “Is each horizontal pod truly limited to one?” Yes, if each pod has 
limit=1
(or 
PREFECT_RUNNER_PROCESS_LIMIT=1
), then that pod will only execute one flow run at a time. But that is not a deployment-wide limit. It is only local to that runner process. 2. “How does the server know there are different serve pods for the same deployment name?” It doesn’t need to track them as named pods in order to schedule correctly. Each pod just polls for the same deployment, and Prefect uses the state transition / claim step to ensure only one pod actually takes a given run. So the server doesn’t need a permanent “runner registry” of your serve pods to avoid duplicate execution. If you want a true cluster-wide limit Use a deployment-level concurrency limit, not just runner 
limit
. In Prefect 3.x, 
limit=
is local runner concurrency. If you want “only 1 run total across all pods for this deployment”, you want the deployment global limit instead. Conceptually:
Copy code
python
my_flow.serve(
    name="my-flow",
    limit=1,         # max 1 per pod
    global_limit=1,  # max 1 across all pods total
)
That gives you: - `limit=1`: each pod can only work on one run at a time - `global_limit=1`: across all pods combined, only one run may be active So: - only 
limit=1
 → one per pod - only 
global_limit=1
 → one total across all pods - both → one per pod, but still one total overall If you want, I can draw the exact behavior for a setup like “3 Kubernetes replicas serving the same deployment” with a timeline example.
o
@Marvin like this? deployment = myflow.to_deployment( name=args.deployment_name, limit=1, global_limit=5, pause_on_shutdown=False, webserver=True ) serve(deployment)
m
✅ thought for 94.2 seconds
Almost — but not quite. If you're building a deployment object first and then calling 
serve(deployment)
, the deployment-level concurrency arg is 
concurrency_limit
, not 
global_limit
. So it should look more like this:
Copy code
python
deployment = myflow.to_deployment(
    name=args.deployment_name,
    concurrency_limit=5,
)

serve(
    deployment,
    limit=1,
    pause_on_shutdown=False,
    webserver=True,
)
Important distinction: - 
limit=1
on 
serve(...)
= runner/pod-local limit - 
concurrency_limit=5
on the deployment = deployment-wide limit across all serve pods So with the above: - each pod can run at most 1 flow run at a time - across all pods serving that same deployment, Prefect allows at most 5 active runs total Why your example is slightly off This version:
Copy code
python
deployment = myflow.to_deployment(
    name=args.deployment_name,
    limit=1,
    global_limit=5,
    pause_on_shutdown=False,
    webserver=True
)
serve(deployment)
is off because: - 
to_deployment(...)
does not take 
limit
- 
to_deployment(...)
does not take 
global_limit
- 
pause_on_shutdown
and 
webserver
belong on 
serve(...)
, not on 
to_deployment(...)
Correct patterns If you want the short single-flow style, this is valid:
Copy code
python
myflow.serve(
    name=args.deployment_name,
    limit=1,
    global_limit=5,
    pause_on_shutdown=False,
    webserver=True,
)
If you want the explicit deployment-object style, use:
Copy code
python
deployment = myflow.to_deployment(
    name=args.deployment_name,
    concurrency_limit=5,
)

serve(
    deployment,
    limit=1,
    pause_on_shutdown=False,
    webserver=True,
)
So the mapping is basically: - 
flow.serve(..., global_limit=5)
↔️ deployment created with 
concurrency_limit=5
- 
flow.serve(..., limit=1)
↔️ 
serve(deployment, limit=1)
If you want, I can also show the Kubernetes replica math for this exact config, e.g. “3 pods, 
limit=1
, 
concurrency_limit=5
-> what actually happens.”
o
@Marvin what does the webserver=True do ?
m
✅ thought for 86.7 seconds
webserver=True
starts a small HTTP server inside the serve process. It is not required to run flows. It’s mainly for health checks and monitoring. What it gives you It exposes a few endpoints like: - 
GET /health
— lets you check if the runner is healthy - 
GET /run_count
— shows how many flow runs are currently executing - 
POST /shutdown
— tells the runner to shut down gracefully So it’s useful for things like: - Kubernetes readiness/liveness probes - container health checks - simple operational monitoring What it does not do - it does not serve the Prefect UI - it does not replace Prefect Cloud / Prefect Server - it is not needed for scheduling or execution Default behavior By default, it’s off unless you set 
webserver=True
. The runner webserver defaults to: - host: 
localhost
- port: 
8080
You can change that with runner server settings/env vars if needed. Practical takeaway If you’re running 
serve(...)
in Kubernetes, 
webserver=True
can be handy so you can point probes at 
/health
. If you don’t need local health endpoints, you can leave it off. Relevant source-backed details were from Prefect 3.x runner/webserver implementation. If helpful, I can show a Kubernetes probe example using 
webserver=True
.
o
@Marvin what i do not get, if one pod replica calcualtes one flow, what is the fuzz about 8o% or 70 % mem utilization. i think your rule is kind of strange. isnt there a better rule to achieve the behavior: we want at least plus 1 serve unused?
m
✅ thought for 20.6 seconds
Yes — if your real goal is: “always keep at least one idle serve pod available” then a memory-utilization rule like “scale at 70–80%” is only an indirect proxy, and it may be a bad one for your workload. If each pod effectively handles one flow run at a time, then the cleanest scaling signal is usually capacity, not memory. Better mental model If each pod has:
Copy code
text
serve(..., limit=1)
then each pod is basically either: - busy = running 1 flow run - idle = running 0 flow runs So the thing you really care about is likely:
Copy code
text
idle pods = total pods - busy pods
And your desired policy is something like:
Copy code
text
idle pods >= 1
That’s much more direct than “memory under 80%”. Why the memory rule feels strange You’re right to question it. Memory-based autoscaling works better when: - load correlates strongly with memory - a pod can process many units of work at once - memory pressure is the bottleneck But in your setup, if a pod runs at most one flow at a time, memory may not reflect scheduling capacity very well: - one pod could be busy but only using 35% memory - another flow could spike memory to 90% - memory says little about whether that pod is available for another run if 
limit=1
So if the point is leave one runner free, memory is not the best primary signal. Better rule for your goal If each pod can run one flow at a time, then you want enough replicas so that:
Copy code
text
replicas >= active_runs + 1
That exactly means “one extra unused serve pod”. Examples: - 
active_runs = 1
-> want 
2
replicas - 
active_runs = 4
-> want 
5
replicas - 
active_runs = 10
-> want 
11
replicas That is a much better rule for your stated behavior. How to think about it operationally Suppose: - every pod runs 
serve(..., limit=1)
- all pods serve the same deployment - deployment concurrency allows enough total runs Then: - 3 replicas = capacity for 3 concurrent runs - if 2 runs are active, 1 pod remains idle - if 3 runs are active, 0 pods remain idle So if you must keep one idle, scale before you hit full occupancy. That means your trigger should be based on busy pods / active flow runs, not memory. Even simpler heuristic If you can observe active runs, the desired replica count is:
Copy code
text
desired_replicas = active_runs + 1
with min/max caps:
Copy code
text
desired_replicas = min(max_replicas, max(min_replicas, active_runs + 1))
What if you cannot scale from active Prefect runs directly? Then better proxies than memory would be: - runner 
/run_count
from 
webserver=True
- number of scheduled / pending / late runs - queue depth / backlog - custom metric: busy runners vs total runners If each serve pod exposes 
/run_count
, then each pod returns either 
0
or 
1
in your setup. Summing that across pods gives actual busy capacity. Then you can scale on:
Copy code
text
desired replicas = sum(run_count) + 1
That’s almost exactly your desired behavior. Important caveat If startup time for a new pod is slow, “+1 idle pod” may still not be enough during bursts. In that case you might want:
Copy code
text
replicas >= active_runs + buffer
where buffer is 
1
, 
2
, or more depending on burstiness and pod startup latency. For example: - slow image pull / cold start -> keep buffer 
2
- fast startup -> buffer 
1
may be enough Bottom line Yes — for your goal, a rule based on 
70–80%
memory utilization is probably not the best rule. A better rule is:
Copy code
text
keep replicas = busy_pods + 1
or more generally:
Copy code
text
keep at least 1 idle pod
That directly matches how 
limit=1
serve pods behave. If you want, I can help translate that into a concrete Kubernetes autoscaling strategy, for example using: - HPA with a custom metric - KEDA - Prometheus metrics from 
/run_count
- or a simpler approximation if you only have CPU/memory available
o
@Marvin what does the community do in this case?
m
✅ thought for 129.7 seconds
Short answer: most people do not solve this with multiple 
serve
pods plus memory-based autoscaling. In the community, the more common pattern for Kubernetes is: use a Kubernetes work pool + worker, and let each flow run launch as its own Kubernetes job/pod. That tends to fit the “keep spare capacity / scale on actual work” goal much better than horizontally scaling 
serve(...)
replicas. What people usually do instead 1. Prefer workers + work pools for Kubernetes This is the standard scalable Kubernetes pattern in Prefect 3.x: - a small number of worker pods poll Prefect - each flow run becomes a new Kubernetes job/pod - Kubernetes handles placement/scaling - cluster/node autoscaling can react to actual pods needing resources That means you don’t need to keep a bunch of long-running 
serve
replicas around just in case. 2. If they use 
serve
, it’s usually for simpler/static setups 
serve(...)
is more of a long-running runner process model: - process stays up - it polls Prefect - it executes runs itself, usually in subprocesses - concurrency is local to that runner That can be fine for: - VM-based deployments - small fixed-capacity services - simple always-on infra - a few regularly running flows But it’s usually not the nicest pattern for “Kubernetes autoscaling with one spare slot”. Why the community tends toward workers in Kubernetes Because your desired behavior is really:
Copy code
text
when new work appears -> create execution capacity
when work is gone -> remove it
That is exactly what Kubernetes jobs + workers are good at. By contrast, with 
serve
replicas you’re trying to infer capacity using: - memory - replica count - local runner limit - idle/busy state That’s workable, but kind of awkward. What people commonly optimize for In Kubernetes, community setups often optimize for one of these: - one flow run = one pod/job - scale cluster nodes when jobs require more resources - set per-deployment resource requests/limits - use deployment/work-pool concurrency limits to cap total active runs That is usually cleaner than: - “3 identical serve pods” - each with 
limit=1
- HPA based on memory - trying to preserve one idle pod So for your exact question — “what does the community do?” Most commonly: - for Kubernetes: use workers + Kubernetes work pools - for simple static infra: use 
serve(...)
- for concurrency control: use deployment-level concurrency limits - for scaling execution: let Kubernetes scale jobs/nodes, not 
serve
replicas based on memory If someone insists on 
serve
replicas Then yes, a better community-style rule would be based on busy vs idle capacity, not memory. Something like:
Copy code
text
desired replicas = active_runs + 1
is much closer to the behavior you want. But that usually requires custom metrics, and once you’re doing that, many teams decide it’s simpler to move to work pools/workers instead. My practical recommendation If your real requirement is: - one run per execution pod - keep some spare capacity - autoscale sensibly in Kubernetes - avoid weird memory heuristics then I would strongly consider switching from 
serve
replicas to a Kubernetes work pool + worker model. That is the more natural and common Prefect-on-Kubernetes pattern. If you want, I can show you a side-by-side of: - your current 
serve
replica model vs - the equivalent Kubernetes work pool setup and explain which one better matches “always have +1 spare”.
4 Views
Open in Slack
PreviousNext
Powered by