https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> in the kubernetes deployment where do...
# ask-marvin
o
@Marvin in the kubernetes deployment where do i adjust the limit and request of cpu and ram of the batch jobs? do i understand it right that i must adjust it per deploymnet? to my understanding the limit and request set for the prefect-worker in kubernetes is not the request and limit for the ephemerial jobs/flows/batch jobs?
m
✅ thought for 220.3 seconds
You're exactly right: the CPU/memory limits you set on the Kubernetes Prefect worker Deployment only apply to the worker pod itself. Each flow run is executed in an ephemeral Kubernetes Job, and you set requests/limits for those jobs via the Kubernetes work pool’s base job template (defaults) and/or per-deployment overrides. How to configure resources for ephemeral jobs (Prefect 3.x): - Work pool (defaults for all deployments using the pool) - Edit the Kubernetes work pool’s Base Job Template (Advanced tab) to include a 
resources
block for the container. Define variables for values you want to override later. - Deployment (override per deployment) - Set 
work_pool.job_variables
in your 
prefect.yaml
or in 
.deploy()
to override the defaults for that specific deployment. - Per run (one-off override) - When triggering a run, pass 
--job-variable
flags or set values in the UI run form. Quick examples 1) Work pool base job template (add variables + reference them in the manifest)
Copy code
{
  "variables": {
    "cpu_request": {
      "type": "string",
      "default": "100m"
    },
    "cpu_limit": {
      "type": "string",
      "default": "1000m"
    },
    "memory_request": {
      "type": "string",
      "default": "256Mi"
    },
    "memory_limit": {
      "type": "string",
      "default": "1Gi"
    }
  },
  "job_configuration": {
    "job_manifest": {
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
                "name": "prefect-job",
                "resources": {
                  "requests": {
                    "cpu": "{{ cpu_request }}",
                    "memory": "{{ memory_request }}"
                  },
                  "limits": {
                    "cpu": "{{ cpu_limit }}",
                    "memory": "{{ memory_limit }}"
                  }
                }
              }
            ]
          }
        }
      }
    }
  }
}
Tip: If your current base template doesn’t include these variables yet, add them as shown and reference them with 
{{ variable_name }}
. 2) Per-deployment override in prefect.yaml
Copy code
deployments:
- name: my-deployment
  entrypoint: flows/my_flow.py:my_flow
  work_pool:
    name: my-k8s-pool
    job_variables:
      cpu_request: "500m"
      cpu_limit: "2"
      memory_request: "512Mi"
      memory_limit: "2Gi"
3) Per-deployment override in code
Copy code
from prefect import flow

@flow
def my_flow():
    ...

flow.from_source(
    source="<https://github.com/myorg/myrepo.git>",
    entrypoint="flows/my_flow.py:my_flow"
).deploy(
    name="my-deployment",
    work_pool_name="my-k8s-pool",
    job_variables={
        "cpu_request": "500m",
        "cpu_limit": "2",
        "memory_request": "512Mi",
        "memory_limit": "2Gi"
    }
)
4) One-off override at run time (CLI)
Copy code
prefect deployment run my-deployment \
  --job-variable cpu_request=250m \
  --job-variable memory_request=128Mi
Helpful commands and docs - Show the default Kubernetes base job template (so you can copy/modify it):
Copy code
prefect work-pool get-default-base-job-template --type kubernetes
- Customize base job templates: https://docs.prefect.io/v3/advanced/customize-base-job-templates - Override job variables on deployments/runs: https://docs.prefect.io/v3/how-to-guides/deployments/customize-job-variables - Run flows on Kubernetes (setup + worker): https://docs.prefect.io/v3/how-to-guides/deployment_infra/kubernetes If you’d like, I can help you edit your current work pool template—just share whether you’re using the UI or CLI and what defaults you want.
o
@Marvin deployment = apply_flow.to_deployment( name=args.deployment_name, concurrency_limit=5 ) # Optionally add webserver=True for a health endpoint serve( limit=1, pause_on_shutdown=False, webserver=True, )
m
✅ thought for 10.3 seconds
It looks like you're mixing deployment creation with serving a flow, and there are a couple of API mismatches for Prefect 3.x. A few important points: * 
serve()
is typically called on the flow object as 
flow.serve(...)
, not as a standalone function with just orchestration options. * 
limit
is not the same as a deployment 
concurrency_limit
* 
webserver=True
is not part of the common 
flow.serve()
signature in Prefect 3.x as far as I’d expect from the public API * If you're using Prefect 3.x, the preferred deployment pattern is 
flow.from_source(...).deploy(...)
or 
flow.deploy(...)
depending on how you're packaging code If you want to serve a flow locally with a concurrency limit, the shape is more like:
Copy code
python
from prefect import flow

@flow
def apply_flow():
    print("hello")

if __name__ == "__main__":
    apply_flow.serve(
        name="my-deployment",
        limit=1,
        pause_on_shutdown=False,
    )
If instead you want to create a deployment object with a deployment-level concurrency limit, that is a different step from serving:
Copy code
python
deployment = apply_flow.to_deployment(
    name=args.deployment_name,
    concurrency_limit=5,
)
Then you would typically serve that deployment or use a worker-based deployment approach depending on your setup. So the main correction to your snippet is that this:
Copy code
python
serve(
    limit=1,
    pause_on_shutdown=False,
    webserver=True,
)
probably should be one of these two patterns: Serve the flow directly:
Copy code
python
apply_flow.serve(
    name=args.deployment_name,
    limit=1,
    pause_on_shutdown=False,
)
Or build a deployment separately if you need deployment configuration:
Copy code
python
deployment = apply_flow.to_deployment(
    name=args.deployment_name,
    concurrency_limit=5,
)
If you want, I can help rewrite your full snippet for either: * local 
serve()
style execution, or * worker / deploy style execution in Prefect 3.x
o
@Marvin yes first i want to create the deployment separately then call serve to serve the deployment with paratmerts like limit and pause on shoutdown.
m
✅ thought for 128.3 seconds
Yes — in Prefect 3.x you can create the deployment separately, then pass that deployment object into the top-level 
serve(...)
. The key distinction is: * 
flow.to_deployment(...)
creates a 
RunnerDeployment
* 
prefect.serve(...)
serves one or more 
RunnerDeployment
objects * you do not call 
.serve()
on the deployment object itself So your pattern should look like this:
Copy code
python
from prefect import flow, serve

@flow
def apply_flow():
    print("running")

deployment = apply_flow.to_deployment(
    name=args.deployment_name,
    concurrency_limit=5,
)

serve(
    deployment,
    limit=1,
    pause_on_shutdown=False,
    webserver=True,
)
A few notes: * 
concurrency_limit=5
on 
to_deployment(...)
is the deployment-level concurrency limit * 
limit=1
on 
serve(...)
is the runner-level limit for how many flow runs this serving process will execute concurrently * 
pause_on_shutdown=False
is valid on 
serve(...)
* 
webserver=True
is supported here as well So if your goal is: 1. create deployment separately 2. serve it with runner options then this is the correct pattern. If you want, I can help turn your exact snippet into a complete 
if __name__ == "__main__":
example with CLI args included.
o
@Marvin Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation. 110438 AM prefect.flow_runs Crash detected! Execution was cancelled by the runtime environment. 110438 AM prefect.flow_runs Finished in state Crashed('Execution was cancelled by the runtime environment.')
m
✅ thought for 109.3 seconds
That message means Prefect lost the concurrency lease for the running flow, so it intentionally crashed the run to avoid exceeding the configured concurrency limit. In practice, this usually means one of these: * the Prefect API/server was temporarily unreachable from the runner/worker * the server restarted and forgot the lease * the lease expired before it could be renewed * you're hitting a known bug around deployment concurrency / retries / timeout behavior Given your earlier example, this is very likely tied to the deployment 
concurrency_limit
 you set in 
to_deployment(...)
. Your log:
Copy code
text
Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation.
Crash detected! Execution was cancelled by the runtime environment.
Finished in state Crashed('Execution was cancelled by the runtime environment.')
means: 1. the run acquired a concurrency slot 2. Prefect tried to renew that slot lease in the background 3. renewal failed because the slot reservation no longer existed 4. Prefect cancelled the run on purpose Common causes * API connectivity issue: intermittent network issue between the serving process and Prefect API * Server restart: if using self-hosted Prefect Server and it restarted, active leases may be lost * Execution lasting past lease-renewal window with trouble renewing * Known bug: there are open issues for this exact message, including retry / timeout related cases: * Issue #20251 * Issue #19068 * Issue #18839 What I’d check first 1. Are you using self-hosted Prefect Server or Prefect Cloud? * if self-hosted: check whether the API/server restarted around 
11:04:38
2. Did the run sit in retry / timeout / long-running state? * one reported bug is specifically around 
AwaitingRetry
3. Was there a network blip between your runner and API? * especially if serving locally, in Docker, or in Kubernetes 4. What Prefect version are you on? * if you're on an older 3.x patch, upgrading is a good idea Most likely mitigation steps * Upgrade Prefect to the latest 3.x patch release first * If this is self-hosted, check API/server logs for restarts or connectivity issues * If the issue started only after adding 
concurrency_limit=5
, try temporarily removing the deployment concurrency limit and see whether the crash disappears * If the run uses retries or long waits, compare against #20251 A good isolation test is:
Copy code
python
deployment = apply_flow.to_deployment(
    name=args.deployment_name,
    # temporarily remove this
    # concurrency_limit=5,
)

serve(
    deployment,
    limit=1,
    pause_on_shutdown=False,
)
If the crashes stop, that strongly suggests the problem is specifically with the deployment concurrency lease path. If you want, send me: * your Prefect version (
prefect version
) * whether you're on Cloud or self-hosted Server * whether the flow has retries / timeout / long sleeps * whether this started only after adding 
concurrency_limit
 and I can help narrow down whether you're hitting infrastructure/connectivity vs a known Prefect bug.
o
@Marvin i am using prefect 3.6.5 and currently see not a possibilty to upgrade. what is the preferred workaround? not setting concurrency of deployment to prenvent the lease issues?
m
✅ thought for 14.3 seconds
Yes — on Prefect 3.6.5, if you cannot upgrade, the most practical workaround is often to avoid deployment-level 
concurrency_limit
 if that is what is triggering the lease-renewal crashes. So in your case, yes, removing:
Copy code
python
deployment = apply_flow.to_deployment(
    name=args.deployment_name,
    concurrency_limit=5,
)
is a reasonable workaround to test first. Then serve it like:
Copy code
python
deployment = apply_flow.to_deployment(
    name=args.deployment_name,
)

serve(
    deployment,
    limit=1,
    pause_on_shutdown=False,
    webserver=True,
)
Important distinction: * 
concurrency_limit
on the deployment uses the lease-based concurrency mechanism that is producing your crash * 
limit=1
on 
serve(...)
limits how many runs that runner process executes concurrently, but it is not the same protection as a deployment concurrency limit across all infrastructure So whether this workaround is acceptable depends on what you need: If you have a single serving process only * using 
serve(..., limit=1)
may be enough as a temporary workaround If you have multiple runners / workers / replicas * removing deployment concurrency means you lose the shared central concurrency enforcement * in that case, multiple processes could still run more flow runs than you want So the preferred workaround order on 3.6.5 would be: 1. Remove deployment 
concurrency_limit
 2. Keep 
serve(..., limit=...)
if a single runner process is sufficient 3. If you need cluster-wide concurrency control, use an external lock / coordination mechanism in your flow logic until you can upgrade 4. Avoid long retry-wait patterns or timeouts combined with deployment concurrency if those are part of your runs Recommended temporary pattern
Copy code
python
from prefect import flow, serve

@flow
def apply_flow():
    print("running")

if __name__ == "__main__":
    deployment = apply_flow.to_deployment(
        name="my-deployment",
    )

    serve(
        deployment,
        limit=1,
        pause_on_shutdown=False,
        webserver=True,
    )
My recommendation If stability is more important than strict central concurrency right now, then yes: remove deployment concurrency and rely on runner-local 
limit
as the workaround. If you want, I can also suggest a safe replacement strategy depending on whether you run: * one local process * multiple containers * Kubernetes * multiple replicas of the same serve process
o
@Marvin in which version the issue was fixed?
m
✅ thought for 206.2 seconds
The best confirmed answer I can give is: The specific fix for this class of lease-renewal issues was merged in PR #21308, which closes: * #20251 * #19068 * #18839 However, I cannot confidently name the exact first release version containing that PR from the data I have here. What I can say confidently: * You are on 
3.6.5
* The issue was still being reported on later versions like 
3.6.8
* There were multiple incremental concurrency-related fixes after 
3.6.5
* The main lease-renewal starvation fix in #21308 is definitely after 
3.6.5
So for your practical question: No, 
3.6.5
does not contain the relevant fix Yes, later 3.6.x releases include important concurrency lease fixes If you cannot upgrade, the preferred workaround is still to avoid deployment 
concurrency_limit
 Relevant links: * PR #21308 * Issue #20251 * Issue #19068 * Issue #18839 * Latest release notes: Prefect releases If you want, I can do one more pass and help infer the likely minimum 3.6.x version from the merge timing / changelog sequence, but I don’t want to overstate certainty without a direct release mapping.
Open in Slack
PreviousNext
Powered by