Short answer: yes—with an automation that detects an OOM crash and re-runs the deployment with higher memory via job variable overrides. Here’s the pattern that works well in Prefect 3.x: - Detect OOM: When a Kubernetes pod is OOMKilled, the flow run is typically marked Crashed and the event text includes “OOMKilled” or “ExitCode 137”. - React: Use an Automation with an Event trigger for flow-run crashed/failed events for your deployment and add a text filter for “OOMKilled” (and/or “ExitCode 137”). - Re-run with more RAM: Add a Run a deployment action that points to the same deployment and overrides job variables to increase memory (e.g., double). Notes and specifics - The “Run a deployment” action supports overriding job variables for the new run (same capability as programmatic

run_deployment(..., job_variables=...)

). Verified callable signature:

run_deployment(
    name,
    parameters=None,
    scheduled_time=None,
    flow_run_name=None,
    timeout=None,
    poll_interval=5,
    tags=None,
    idempotency_key=None,
    work_queue_name=None,
    as_subflow=True,
    job_variables=None,
)

- If you already pass CPU/RAM to your Kubernetes worker via job variables, you can supply new values in the automation action for the retry. Example job variables JSON you’d put in the action:

{
  "cpu": "2",
  "memory": "8Gi"
}

Recommended guardrails - Prevent loops: Add a distinguishing tag or parameter on the “escalated” run (e.g., add tag

oom-retry

) and include a trigger filter that excludes runs with that tag so you don’t escalate repeatedly. - Preserve parameters: In the action, set parameters to the same ones from the failed run. If you prefer templates, you can copy the triggering run’s parameters; if not, just leave parameters empty and rely on your deployment defaults. - Escalation tiers: Because “double” may require templating, many teams set up one or more tiers (e.g., default → high-mem) and have the automation run a “high-mem” variant of the deployment. This avoids arithmetic in templates and is easy to reason about. Two implementation options 1) Single deployment; override job variables in the action - Trigger: Flow run crashed (text contains “OOMKilled”). - Action: Run deployment (same deployment), set: - Tags: add

oom-retry

- Job variables: set a larger memory value (e.g., from

4Gi

to

8Gi

). 2) Two deployments (default and high-mem) - Trigger: Flow run crashed (text contains “OOMKilled”) AND deployment == default - Action: Run deployment → pick the high-mem deployment (with doubled RAM baked in) - Filter the trigger to not fire on high-mem deployment runs. Questions so I can tailor exact steps for you - Are you on Prefect Cloud or Prefect Server? (Creating automations is usually done in the Cloud UI; the local CLI can list/pause/resume/delete but not create.) - What job variable keys are you using for CPU/RAM today (e.g.,

cpu

,

memory

,

pod_override

, etc.)? I can provide the exact JSON you’d use in the action. - Do you want to escalate once (to a single higher tier) or keep escalating across tiers? If you’d like, share your current deployment’s job variables format and I’ll draft the precise automation trigger and action payload you can drop in.

Great — since you’re on self-hosted Prefect Server and you already pass a

resources

dict as a job variable (mirroring Kubernetes requests/limits), you can handle OOM restarts with an Automation that: 1) Detects an OOM crash 2) Re-runs the same deployment 3) Overrides

job_variables.resources

with higher memory How to set it up - In the UI, create a new Automation: - Trigger: Flow run enters state Crashed (you can also include Failed if you want), and add a text filter like “OOMKilled” or “ExitCode 137”. - Filter the trigger to only apply to the target deployment(s). - Add a filter to exclude runs with a tag like

oom-retry

(see loop prevention below). - Action: Run a deployment - Deployment: the same deployment - Tags to add to the new run:

oom-retry

- Job variables: provide a higher-memory

resources

dict. For example, if your baseline was 4Gi, bump to 8Gi:

{
  "resources": {
    "requests": {
      "cpu": "1",
      "memory": "8Gi"
    },
    "limits": {
      "cpu": "1",
      "memory": "8Gi"
    }
  }
}

- Parameters: leave blank if you want the deployment defaults, or set explicitly if you need to preserve specific params. Notes and guardrails - Loop prevention: Because the escalated run might also OOM, add tag

oom-retry

in the action and add a trigger filter that excludes runs with the

oom-retry

tag. That way, only the first OOM will escalate. - Fixed tiers vs “double”: Automations don’t “compute” values, so the simplest approach is to define one or more fixed tiers (e.g., default → high-mem). If you truly want “double,” you can approximate it by: - Maintaining two deployments (default and high-mem) and have the automation run the high-mem one; or - Using a static override in the action that you know is roughly 2x your baseline. - Merge behavior: The job variables you set in the action will override/merge with your deployment’s job variables. Make sure you keep the same structure your worker template expects (i.e.,

job_variables.resources.requests/limits

). - What to match on for OOM: In practice, OOMKilled pods will yield a Crashed flow run and logs that include “OOMKilled” or “ExitCode 137”; match on those in the automation’s text filter. If you share your current baseline

resources

dict, I can suggest the exact override JSON (2x RAM, same CPU, or both) and the precise trigger filters you should use.