<@ULVA73B9P> I have a flow run that runs a subflow...
# ask-marvin
s
@Marvin I have a flow run that runs a subflow:
Copy code
def outer_flow():
   inner_flow.with_options(
        persist_result=True,
        on_cancellation=[cancellation_callback]
   )
If I cancel the outer flow from the Prefect UI, it seems like the on_cancellation hook for the inner_flow subflow does not run. Why is this? How can I ensure that a subflow's on_cancellation hook runs when it's parent flow is cancelled?
m
thought for 558.2 seconds
Short answer - What you’re seeing is expected. In-process subflows are set directly to Cancelled when their parent is cancelled, so the subflow never enters the Cancelling state and its
on_cancellation
hooks do not run. - To guarantee a subflow’s
on_cancellation
hook runs when its parent is cancelled, run the subflow as its own deployment and call it via
run_deployment
so it has an independent runner/process. Deployed subflows enter Cancelling first, and their
on_cancellation
hooks are executed. Why this happens - In Prefect 3.x, parent→child cancellation is handled by a server-side cleanup service. - For subflows that run in the same process as the parent (your case), when the parent is cancelled the server will mark the subflow directly as Cancelled. Prefect only executes
on_cancellation
hooks when a run transitions to Cancelling; going straight to Cancelled skips those hooks. - Deployed flow runs (including deployed subflows) are transitioned to Cancelling first; the runner sees that state and executes
on_cancellation
before final Cancelled. How to ensure the subflow hook runs Option A: Make the subflow a deployment and call it by deployment 1) Create a deployment for the inner flow (from your project directory):
Copy code
prefect deploy ./path/to/file.py:inner_flow -n inner-deployment
2) Call it from the parent via
run_deployment
so it runs independently:
Copy code
from prefect import flow
from prefect.deployments import run_deployment

def cancellation_callback(flow, flow_run, state):
    print("inner cleanup")

@flow(on_cancellation=[cancellation_callback])
def inner_flow():
    ...

@flow
def outer_flow():
    # Triggers an independent run that will enter Cancelling on parent cancel
    run_deployment("inner-flow/inner-deployment")
Notes: - There can be a short delay (typically up to ~20–30s) between cancelling the parent and the subflow entering Cancelling due to server polling intervals. Option B: Keep it in-process but don’t rely on on_cancellation - Put critical cleanup in a
finally
block inside the subflow so it runs during a graceful shutdown:
Copy code
from prefect import flow

@flow
def inner_flow():
    try:
        # work
        ...
    finally:
        # best-effort cleanup even if parent is cancelled
        ...
- Or move the cleanup to the parent’s
on_cancellation
(if it can safely clean up the child’s resources). Key takeaways -
on_cancellation
hooks only run when a run enters Cancelling. - In-process subflows go straight to Cancelled on parent cancellation, so their
on_cancellation
hooks won’t run. - Use a separate deployment +
run_deployment
if you need the subflow’s
on_cancellation
hook to run reliably.
n
hmm. i will try a repro for this since i guess i have the same intuition as you here
thank you 1
s
thanks, that would be super helpful. Maybe a total shot in the dark, but I noticed that the infrastructure PID is unset for my subflow although it is set for the parent. Could this be related?
@Marvin I can see that my subflow does enter a Cancelling state before proceeding to Cancelled, so you might be wrong about something
m
thought for 241.9 seconds
Great catch — you’re right that a subflow can show Cancelling before Cancelled. Thanks for calling that out. What’s going on - Server state vs execution context: When a parent is cancelled, the server’s cancellation cleanup service updates child runs. If the subflow run has a deployment_id, the service puts it into Cancelling first; otherwise it can go straight to Cancelled. - Source: cancellation_cleanup.py - Related policy: BypassCancellingFlowRunsWithNoInfra - Why your subflow shows Cancelling but the hook still doesn’t run: even if the server marks a subflow as Cancelling, the subflow’s on_cancellation hooks only run if there’s a runner “owning” that subflow run to observe the state and execute hooks before shutting down. When the inner flow is executed in-process (called directly inside the parent), it does not have a dedicated runner; the parent runner tears down the whole process and the inner subflow doesn’t get a chance to execute its own hooks. How to make the subflow’s on_cancellation hook run reliably - Best practice: run the inner flow as its own deployment so it has its own runner/process. Then call it from the parent via
run_deployment
. The subflow will enter Cancelling, its runner will see that state, and the on_cancellation hooks will execute before the final Cancelled state. 1) Create the inner deployment:
Copy code
prefect deploy ./path/to/file.py:inner_flow -n inner-deployment
2) Call it from the parent:
Copy code
from prefect import flow
  from prefect.deployments import run_deployment

  def cancellation_callback(flow, flow_run, state):
      print("inner cleanup")

  @flow(on_cancellation=[cancellation_callback])
  def inner_flow(...):
      ...

  @flow
  def outer_flow(...):
      # call as an independent run
      run_deployment("inner-flow/inner-deployment", parameters={...})
- Note: there can be a short delay (often up to ~20–30s) between cancelling the parent and the subflow entering Cancelling due to server polling intervals. - If you must keep it in-process: don’t rely on
on_cancellation
for the inner flow. Put critical cleanup in a finally block (best-effort) or centralize cleanup in the parent’s
on_cancellation
.
Copy code
from prefect import flow

  @flow
  def inner_flow():
      try:
          ...
      finally:
          # best-effort cleanup even if parent is cancelled
          ...
If you can share how you’re invoking the inner flow today (direct call vs
run_deployment
), I can suggest the smallest change to get your hook to fire.
n
coming back to this, i think you and marvin are roughly correct. in-process subflows are sigtermed when their parent is cancelled, so they go to crashed. if you moved what you have in your cancel hooks to crash hooks itd run if that's not ideal for some specific reason i thnk this would be a great thing to codify as a discussion