<@ULVA73B9P> Why flow ends in Cancelled state ? Th...
# ask-marvin
r
@Marvin Why flow ends in Cancelled state ? This error was logged :
Copy code
Jun 18th, 2026
Worker 'ProcessWorker ee406fd7-8d70-4135-95c6-40ac4344a8d2' submitting flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd'
09:33:06 AM
prefect.flow_runs.worker
Aborted submission of flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd': Server returned a non-pending state 'SCHEDULED'
09:33:06 AM
prefect.flow_runs.worker
Worker 'ProcessWorker ee406fd7-8d70-4135-95c6-40ac4344a8d2' submitting flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd'
09:33:32 AM
prefect.flow_runs.worker
Aborted submission of flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd': Server returned a non-pending state 'SCHEDULED'
09:33:32 AM
prefect.flow_runs.worker
Worker 'ProcessWorker ee406fd7-8d70-4135-95c6-40ac4344a8d2' submitting flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd'
09:33:57 AM
prefect.flow_runs.worker
Aborted submission of flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd': Server returned a non-pending state 'SCHEDULED'
09:33:58 AM
prefect.flow_runs.worker
Worker 'ProcessWorker ee406fd7-8d70-4135-95c6-40ac4344a8d2' submitting flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd'
09:34:26 AM
prefect.flow_runs.worker
Starting flow run process...
09:34:26 AM
prefect.flow_runs.runner
Flow run 'fluffy-saiga' submitted to infrastructure
09:34:26 AM
prefect.flow_runs.worker
Running 1 deployment pull step(s)
09:34:28 AM
prefect.flow_runs
Executing deployment step: set_working_directory
09:34:28 AM
prefect.flow_runs
Deployment step 'set_working_directory' completed successfully
09:34:28 AM
prefect.flow_runs
All deployment steps completed successfully
09:34:28 AM
prefect.flow_runs
Changing working directory to '/apphome/clag'
09:34:28 AM
prefect.flow_runs
Importing flow code from 'lif_snowflake.oracle2snowflake_cdt:flow_ora2sw_cdt'
09:34:28 AM
prefect.flow_runs
Engine execution exited with unexpected exception
Traceback (most recent call last):
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 2191, in run_flow
    ret_val = run_flow_sync(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 2036, in run_flow_sync
    return engine.state if return_type == "state" else engine.result()
                                                       ^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 776, in result
    _result = self.state.result(raise_on_failure=raise_on_failure, _sync=True)  # type: ignore
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/compatibility/async_dispatch.py", line 94, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/client/schemas/objects.py", line 394, in result
    return run_coro_as_sync(
           ^^^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/utilities/asyncutils/__init__.py", line 207, in run_coro_as_sync
    return call.result()
           ^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/utilities/asyncutils/__init__.py", line 188, in coroutine_wrapper
    return await task
           ^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/states.py", line 85, in get_state_result
    return await _get_state_result(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/states.py", line 157, in _get_state_result
    raise await aget_state_exception(state)
prefect.exceptions.CancelledRun: Deployment concurrency slot lost during provisioning - no slots available to continue execution
m
thought for 131.2 seconds
That log points to deployment concurrency, not a user-triggered cancellation. Why it ended as
Cancelled
Your flow successfully got picked up by the worker, but Prefect cancelled it before actual execution because it lost its deployment concurrency slot during provisioning. The key line is:
Copy code
text
prefect.exceptions.CancelledRun: Deployment concurrency slot lost during provisioning - no slots available to continue execution
What that means in practice: 1. The flow run was trying to start under a deployment with a
concurrency_limit
. 2. Prefect reserved a slot for it while the worker was provisioning/ramping up the run. 3. Before the run reached
Running
, that reservation expired or was no longer valid. 4. When Prefect checked again, no slot was available anymore. 5. Since the run had already begun provisioning, Prefect cancelled it. So the
Cancelled
state is expected behavior for that concurrency protection path. What the
SCHEDULED
messages mean
These worker logs:
Copy code
text
Aborted submission of flow run ...: Server returned a non-pending state 'SCHEDULED'
usually mean the worker tried to submit the run, but the API told it the run was no longer in a state that could be started yet. In concurrency scenarios, Prefect may bounce the run back into scheduling/orchestration while it waits for a valid slot. That message is more of a symptom of orchestration/concurrency timing, not the root cause. The root cause is the final exception about losing the deployment concurrency slot during provisioning. Most likely causes Common reasons this happens: - Your deployment has a low
concurrency_limit
- Multiple runs are competing for the same deployment slot - Startup/provisioning takes long enough that the slot lease expires before the flow reaches
Running
- Another run grabs the freed slot first How to fix it 1. Increase the deployment grace period If your infrastructure startup is slow, give the concurrency lease more time. 2. Use
ENQUEUE
collision strategy instead of cancel-style behavior
That makes runs wait for a slot instead of getting cancelled. 3. Reduce startup/provisioning time Faster worker/infrastructure startup reduces the chance of losing the slot. 4. Increase the deployment concurrency limit If appropriate for your workload. 5. Upgrade Prefect/worker Newer 3.x versions have improvements around concurrency lease handling. If you're not on a recent version, upgrade both server and workers. What to inspect next Please check whether this deployment has something like: -
concurrency_limit
-
concurrency_options
- a collision strategy - slow startup steps before the flow begins running If you share your deployment config, I can tell you exactly what to change. Example direction If you define deployments in Python, the fix often looks like increasing concurrency grace time and enqueueing:
Copy code
python
from prefect.client.schemas.objects import ConcurrencyOptions, ConcurrencyLimitStrategy

flow.from_source(...).deploy(
    name="my-deployment",
    work_pool_name="my-pool",
    concurrency_limit=5,
    concurrency_options=ConcurrencyOptions(
        collision_strategy=ConcurrencyLimitStrategy.ENQUEUE,
        grace_period_seconds=900,
    ),
)
If you use
prefect.yaml
, it would be the same idea in deployment settings. Useful docs - Prefect deployments: Deployments -
prefect deploy
CLI: prefect deploy - Work pools/workers: Work pools and workers If you want, paste your deployment definition or
prefect.yaml
and I’ll point out the exact setting causing this.
r
@Marvin What is default value for grace_period_seconds ?
m
thought for 6.2 seconds
The default is 300 seconds (
5 minutes
). So if you do not set
grace_period_seconds
, Prefect gives the deployment concurrency lease 5 minutes during provisioning before it can be considered lost. If your run startup can take longer than that, set a higher value explicitly. Example:
Copy code
python
from prefect.client.schemas.objects import ConcurrencyOptions, ConcurrencyLimitStrategy

flow.from_source(...).deploy(
    name="my-deployment",
    work_pool_name="my-pool",
    concurrency_limit=5,
    concurrency_options=ConcurrencyOptions(
        collision_strategy=ConcurrencyLimitStrategy.ENQUEUE,
        grace_period_seconds=900,
    ),
)
If you want, I can also show where to set this in
prefect.yaml
.
r
@Marvin I got
prefect.exceptions.CancelledRun: Deployment concurrency slot lost during provisioning - no slots available to continue execution
But it does not take 5 minutes ... Here is the log :
Copy code
Worker 'ProcessWorker ee406fd7-8d70-4135-95c6-40ac4344a8d2' submitting flow run '019ed936-2ebc-730b-ba5f-dcc6be5836a4'
08:33:30 AM
prefect.flow_runs.worker
Starting flow run process...
08:33:30 AM
prefect.flow_runs.runner
Flow run 'watchful-barnacle' submitted to infrastructure
08:33:30 AM
prefect.flow_runs.worker
Running 1 deployment pull step(s)
08:33:31 AM
prefect.flow_runs
Executing deployment step: set_working_directory
08:33:31 AM
prefect.flow_runs
Deployment step 'set_working_directory' completed successfully
08:33:31 AM
prefect.flow_runs
All deployment steps completed successfully
08:33:31 AM
prefect.flow_runs
Changing working directory to '/apphome/clag'
08:33:31 AM
prefect.flow_runs
Importing flow code from 'scanner.main:simple_scanner'
08:33:31 AM
prefect.flow_runs
Engine execution exited with unexpected exception
Traceback (most recent call last):
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 2191, in run_flow
    ret_val = run_flow_sync(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 2036, in run_flow_sync
    return engine.state if return_type == "state" else engine.result()
                                                       ^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 776, in result
    _result = self.state.result(raise_on_failure=raise_on_failure, _sync=True)  # type: ignore
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/compatibility/async_dispatch.py", line 94, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/client/schemas/objects.py", line 394, in result
    return run_coro_as_sync(
           ^^^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/utilities/asyncutils/__init__.py", line 207, in run_coro_as_sync
    return call.result()
           ^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/utilities/asyncutils/__init__.py", line 188, in coroutine_wrapper
    return await task
           ^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/states.py", line 85, in get_state_result
    return await _get_state_result(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/states.py", line 157, in _get_state_result
    raise await aget_state_exception(state)
prefect.exceptions.CancelledRun: Deployment concurrency slot lost during provisioning - no slots available to continue execution
08:33:32 AM
prefect.flow_runs
Process for flow run 'watchful-barnacle' exited with status code: 1; Process exited with a general error.
08:33:32 AM
prefect.flow_runs.runner
Check the flow run logs for an unhandled exception or assertion error.
08:33:32 AM
prefect.flow_runs.runner
Check the flow run logs for an unhandled exception or assertion error.
m
thought for 127.8 seconds
Yes — this can happen in much less than 5 minutes. The
300s
default is the lease duration, but this error does not strictly mean “the full 300 seconds elapsed.” It means that when Prefect tried to move the run from
Pending
to
Running
, the concurrency lease was already gone and no slot was available to reacquire. In your log, that check is happening almost immediately after import starts:
Copy code
text
08:33:30 submitting flow run
08:33:30 Starting flow run process...
08:33:31 Importing flow code ...
08:33:31 Engine execution exited ...
08:33:32 CancelledRun: Deployment concurrency slot lost during provisioning
So this is not “startup took more than 5 minutes.” It’s more likely one of these: Most likely causes 1. Another competing run took the slot - Your run had a slot while being provisioned - By the time Prefect validated the
Pending -> Running
transition, that slot was no longer valid - Another run for the same deployment may have acquired it 2. Worker/server version mismatch - This is a known thing to check with deployment concurrency leasing - If the worker is older and the server is newer, lease renewal/validation behavior can be inconsistent 3. A very small custom grace period is actually configured - Even if you expected the default, the deployment may explicitly set a lower value 4. Concurrency lease/revocation race - The run can lose the lease due to orchestration timing, even very quickly, if the server decides the original lease is no longer valid and cannot reacquire a slot Important point This message does not mean: - your code import failed - your flow function is broken - Prefect waited 5 minutes It means: - Prefect cancelled the run before execution could continue, because deployment concurrency enforcement rejected the final transition to
Running
. What I’d check first 1. Check Prefect versions on both worker and server/API - especially whether the worker is older than the server - I strongly recommend using a recent Prefect 3.x on both sides 2. Check deployment concurrency settings -
concurrency_limit
-
concurrency_options
-
grace_period_seconds
- collision strategy 3. Check whether multiple runs of the same deployment were starting together - if limit is
1
, this becomes much easier to hit 4. Switch collision strategy to
ENQUEUE
- this is often safer than cancellation when there is contention Likely fix If this is a deployment with concurrency control, I’d try:
Copy code
python
from prefect.client.schemas.objects import ConcurrencyOptions, ConcurrencyLimitStrategy

flow.from_source(...).deploy(
    name="my-deployment",
    work_pool_name="my-pool",
    concurrency_limit=1,
    concurrency_options=ConcurrencyOptions(
        collision_strategy=ConcurrencyLimitStrategy.ENQUEUE,
        grace_period_seconds=900,
    ),
)
That helps in two ways: - longer lease during provisioning - queue/wait behavior instead of canceling when slots are contested What would help narrow it down Can you share: - your Prefect version on the worker - your Prefect server/cloud version if self-hosted - the deployment’s concurrency config - whether
concurrency_limit=1
for that deployment - whether multiple runs were triggered around the same time If you paste the deployment definition or
prefect.yaml
, I can tell you exactly which setting is causing this.
r
@Marvin Worker version: 3.7.1 API version: 3.7.1 (self hosted) Background services version: 3.7.1 (self hosted) Deployment configuration :
Copy code
schedules=[
            DeploymentScheduleCreate(
                schedule=IntervalSchedule(interval=timedelta(minutes=5)),
                active=True,
                max_scheduled_runs=1,
            )
        ],
        concurrency_limit=1,
        concurrency_options=ConcurrencyOptions(
            collision_strategy=ConcurrencyLimitStrategy.CANCEL_NEW, # I don't want to plan next run, if old one is running
        ),
Only this run was triggered. Scheduling is set to every 5 minutes.
m
thought for 13.5 seconds
Thanks — that helps a lot. Given: - worker
3.7.1
- API
3.7.1
- background services
3.7.1
-
concurrency_limit=1
-
collision_strategy=CANCEL_NEW
- schedule every 5 minutes - only one visible run triggered this should not be a version mismatch issue. What’s most likely happening is: Your deployment concurrency limit is 1, and at the moment this run transitions to
Running
, Prefect thinks the only slot is already occupied.
With
CANCEL_NEW
, Prefect will cancel instead of queueing/retrying if it cannot keep or reacquire the slot. That matches your error exactly. Why this can happen even with “only one run” There are a few realistic explanations: 1. A previous run still held the deployment slot briefly - even if the prior run looked finished, cancellation/finalization/cleanup may not have fully released the slot yet - with a 5-minute schedule and
concurrency_limit=1
, a boundary race is possible 2. The same run temporarily lost its lease and then failed to reacquire it - at
Pending -> Running
, Prefect validates the deployment slot again - if lease validation fails and slot count says
0 available
, the run is cancelled 3. There may be another run attempt/reschedule not obvious from the UI at first glance - especially around scheduled runs, retries, or a run that was created then rescheduled/cancelled quickly 4.
CANCEL_NEW
makes this behavior harsher
- even a transient slot conflict leads to cancellation instead of waiting Most important insight With:
Copy code
python
concurrency_limit=1
collision_strategy=ConcurrencyLimitStrategy.CANCEL_NEW
you are effectively telling Prefect:
“If anything else appears to hold the deployment slot, cancel the new run immediately.”
So even a short-lived slot accounting conflict can produce exactly what you saw. What I recommend Option 1: Switch to
ENQUEUE
This is the safest fix if you want to eliminate these transient cancellations.
Copy code
python
concurrency_options=ConcurrencyOptions(
    collision_strategy=ConcurrencyLimitStrategy.ENQUEUE,
)
That does not allow overlap. It just means: “wait until the old run is really done” instead of canceling. If your concern is “I don’t want stacked future runs,”
ENQUEUE
may still be acceptable if you also keep scheduling tight and limit what gets created. Option 2: Keep
CANCEL_NEW
, but expect occasional cancellations
If you truly want “drop the new run if the old one is still in any way active,” then this behavior is consistent with that policy. Option 3: Add a larger grace period This may reduce lease-loss edge cases:
Copy code
python
concurrency_options=ConcurrencyOptions(
    collision_strategy=ConcurrencyLimitStrategy.CANCEL_NEW,
    grace_period_seconds=900,
)
But I suspect your main issue is less about lease duration and more about
CANCEL_NEW
with
concurrency_limit=1
. My read of your case Because the failure happens within ~2 seconds, I do not think: - the run exceeded the default 5-minute grace period - worker/API version mismatch is involved I do think: - the deployment slot was not available at the exact orchestration check - and
CANCEL_NEW
caused immediate cancellation rather than waiting Suggested next step I’d try this first:
Copy code
python
concurrency_limit=1,
concurrency_options=ConcurrencyOptions(
    collision_strategy=ConcurrencyLimitStrategy.ENQUEUE,
    grace_period_seconds=300,
)
That still preserves no parallel runs, but avoids cancellation due to a tiny timing race. If you really need “skip rather than wait,” then
CANCEL_NEW
is the right semantic choice — but then these occasional cancellations are an expected side effect when slot availability is ambiguous for even a moment. One more thing to inspect Check whether the previous scheduled run was still in a finalizing state around
08:33:30
, even if it looked done. With a 5-minute interval and limit
1
, that’s the most plausible trigger.
If you want, I can help you design a setup that means: - never overlap runs - never queue a backlog - skip a run if the previous one is still active because that is subtly different from raw deployment concurrency, and there may be a better pattern for it.
r
@Marvin But why concurrency slot was not available ? Last 3 runs ended in Cancelled state. They we scheluled at 8:23, 8:28, 8:33 and all ended with that error after 2-3 sec ...
m
thought for 11.8 seconds
That pattern is the key. If every scheduled run at
8:23
,
8:28
,
8:33
is cancelled within 2–3 seconds with:
Copy code
text
Deployment concurrency slot lost during provisioning - no slots available to continue execution
then this is probably not real contention from another active run. With
concurrency_limit=1
, repeated immediate cancellations strongly suggest the deployment’s concurrency slot is effectively stuck as occupied or Prefect’s slot/lease state is inconsistent. What’s likely happening Prefect is doing something like: 1. schedule creates the run 2. worker starts provisioning it 3. when transitioning
Pending -> Running
, Prefect checks the deployment concurrency slot 4. Prefect believes no slot is available 5. because strategy is
CANCEL_NEW
, the run is cancelled immediately If that happens for every run, then one of these is likely true: 1. A stale active slot is stuck on the deployment concurrency limit - Prefect thinks one run is still holding the slot - but that run is no longer actually running 2. A stale/orphaned lease exists - the lease backing deployment concurrency was not cleaned up correctly - new runs cannot reacquire the slot 3. There’s a server-side state inconsistency/bug around deployment concurrency - especially since all runs fail the same way immediately Because this is self-hosted, this is much more plausible than “another hidden run is always starting at the exact same time.” Why the first cancelled run can poison the next ones If one run loses the slot during provisioning and cleanup does not fully release the slot, then the next scheduled run sees: - slot already occupied -
CANCEL_NEW
=> cancel - cleanup again may not clear it - repeat forever That would explain the exact loop you’re seeing every 5 minutes. What to check right now Please inspect the deployment’s concurrency state in the UI/API and look for: - active slot count =
1
- no actual running flow runs If that’s true, you’ve basically confirmed a stuck concurrency slot. Fastest test The quickest way to verify this is the deployment concurrency state itself. Also, a very useful test is: temporarily remove deployment concurrency Set:
Copy code
python
concurrency_limit=None
or redeploy without
concurrency_limit
. If the next run starts normally, then you’ve confirmed the issue is specifically the deployment concurrency slot state, not your flow code or worker. Second test Change only this:
Copy code
python
collision_strategy=ConcurrencyLimitStrategy.ENQUEUE
If the run no longer gets immediately cancelled, that also confirms the failure is happening at the concurrency check rather than flow startup. Most likely root cause in your case Based on the repeated 5-minute failures, I think the most likely explanation is: A stale deployment concurrency slot / orphaned lease is stuck after an earlier cancelled run, so every new scheduled run sees the slot as unavailable and gets cancelled by
CANCEL_NEW
.
What to do Best immediate workaround 1. redeploy without deployment concurrency, or 2. switch to
ENQUEUE
Best diagnostic step - check whether the deployment concurrency slot count says occupied while no run is running If you want skip-not-queue semantics Deployment concurrency with
CANCEL_NEW
may not be the best fit for “run every 5 minutes, but silently skip if previous run is still active.” It can be brittle if slot accounting gets stuck. A more robust pattern is often: - allow scheduling - at flow start, check whether a previous run is still active - if yes, exit cleanly with
Completed("skipped")
or similar That avoids dependency on deployment slot leasing for skip semantics. Docs - Deployments: Deployments - Work pools/workers: Work pools and workers If you want, I can help you with either:
1. how to inspect/reset the stuck deployment concurrency state, or 2. how to implement “skip if previous still running” without using
CANCEL_NEW
.