Robert Szomolanyi
06/18/2026, 7:48 AMJun 18th, 2026
Worker 'ProcessWorker ee406fd7-8d70-4135-95c6-40ac4344a8d2' submitting flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd'
09:33:06 AM
prefect.flow_runs.worker
Aborted submission of flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd': Server returned a non-pending state 'SCHEDULED'
09:33:06 AM
prefect.flow_runs.worker
Worker 'ProcessWorker ee406fd7-8d70-4135-95c6-40ac4344a8d2' submitting flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd'
09:33:32 AM
prefect.flow_runs.worker
Aborted submission of flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd': Server returned a non-pending state 'SCHEDULED'
09:33:32 AM
prefect.flow_runs.worker
Worker 'ProcessWorker ee406fd7-8d70-4135-95c6-40ac4344a8d2' submitting flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd'
09:33:57 AM
prefect.flow_runs.worker
Aborted submission of flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd': Server returned a non-pending state 'SCHEDULED'
09:33:58 AM
prefect.flow_runs.worker
Worker 'ProcessWorker ee406fd7-8d70-4135-95c6-40ac4344a8d2' submitting flow run 'c3d672ff-f00c-44fe-a2b4-34d55d453fdd'
09:34:26 AM
prefect.flow_runs.worker
Starting flow run process...
09:34:26 AM
prefect.flow_runs.runner
Flow run 'fluffy-saiga' submitted to infrastructure
09:34:26 AM
prefect.flow_runs.worker
Running 1 deployment pull step(s)
09:34:28 AM
prefect.flow_runs
Executing deployment step: set_working_directory
09:34:28 AM
prefect.flow_runs
Deployment step 'set_working_directory' completed successfully
09:34:28 AM
prefect.flow_runs
All deployment steps completed successfully
09:34:28 AM
prefect.flow_runs
Changing working directory to '/apphome/clag'
09:34:28 AM
prefect.flow_runs
Importing flow code from 'lif_snowflake.oracle2snowflake_cdt:flow_ora2sw_cdt'
09:34:28 AM
prefect.flow_runs
Engine execution exited with unexpected exception
Traceback (most recent call last):
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 2191, in run_flow
ret_val = run_flow_sync(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 2036, in run_flow_sync
return engine.state if return_type == "state" else engine.result()
^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 776, in result
_result = self.state.result(raise_on_failure=raise_on_failure, _sync=True) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/compatibility/async_dispatch.py", line 94, in wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/client/schemas/objects.py", line 394, in result
return run_coro_as_sync(
^^^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/utilities/asyncutils/__init__.py", line 207, in run_coro_as_sync
return call.result()
^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
return self.future.result(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
result = await coro
^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/utilities/asyncutils/__init__.py", line 188, in coroutine_wrapper
return await task
^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/states.py", line 85, in get_state_result
return await _get_state_result(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/states.py", line 157, in _get_state_result
raise await aget_state_exception(state)
prefect.exceptions.CancelledRun: Deployment concurrency slot lost during provisioning - no slots available to continue executionMarvin
06/18/2026, 7:49 AMMarvin
06/18/2026, 7:51 AMCancelled
Your flow successfully got picked up by the worker, but Prefect cancelled it before actual execution because it lost its deployment concurrency slot during provisioning.
The key line is:
text
prefect.exceptions.CancelledRun: Deployment concurrency slot lost during provisioning - no slots available to continue execution
What that means in practice:
1. The flow run was trying to start under a deployment with a concurrency_limit.
2. Prefect reserved a slot for it while the worker was provisioning/ramping up the run.
3. Before the run reached Running, that reservation expired or was no longer valid.
4. When Prefect checked again, no slot was available anymore.
5. Since the run had already begun provisioning, Prefect cancelled it.
So the Cancelled state is expected behavior for that concurrency protection path.
What the SCHEDULED messages mean
These worker logs:
text
Aborted submission of flow run ...: Server returned a non-pending state 'SCHEDULED'
usually mean the worker tried to submit the run, but the API told it the run was no longer in a state that could be started yet. In concurrency scenarios, Prefect may bounce the run back into scheduling/orchestration while it waits for a valid slot.
That message is more of a symptom of orchestration/concurrency timing, not the root cause.
The root cause is the final exception about losing the deployment concurrency slot during provisioning.
Most likely causes
Common reasons this happens:
- Your deployment has a low concurrency_limit
- Multiple runs are competing for the same deployment slot
- Startup/provisioning takes long enough that the slot lease expires before the flow reaches Running
- Another run grabs the freed slot first
How to fix it
1. Increase the deployment grace period
If your infrastructure startup is slow, give the concurrency lease more time.
2. Use ENQUEUE collision strategy instead of cancel-style behavior
That makes runs wait for a slot instead of getting cancelled.
3. Reduce startup/provisioning time
Faster worker/infrastructure startup reduces the chance of losing the slot.
4. Increase the deployment concurrency limit
If appropriate for your workload.
5. Upgrade Prefect/worker
Newer 3.x versions have improvements around concurrency lease handling. If you're not on a recent version, upgrade both server and workers.
What to inspect next
Please check whether this deployment has something like:
- concurrency_limit
- concurrency_options
- a collision strategy
- slow startup steps before the flow begins running
If you share your deployment config, I can tell you exactly what to change.
Example direction
If you define deployments in Python, the fix often looks like increasing concurrency grace time and enqueueing:
python
from prefect.client.schemas.objects import ConcurrencyOptions, ConcurrencyLimitStrategy
flow.from_source(...).deploy(
name="my-deployment",
work_pool_name="my-pool",
concurrency_limit=5,
concurrency_options=ConcurrencyOptions(
collision_strategy=ConcurrencyLimitStrategy.ENQUEUE,
grace_period_seconds=900,
),
)
If you use prefect.yaml, it would be the same idea in deployment settings.
Useful docs
- Prefect deployments: Deployments
- prefect deploy CLI: prefect deploy
- Work pools/workers: Work pools and workers
If you want, paste your deployment definition or prefect.yaml and I’ll point out the exact setting causing this.Robert Szomolanyi
06/18/2026, 10:10 AMMarvin
06/18/2026, 10:10 AMMarvin
06/18/2026, 10:10 AM5 minutes).
So if you do not set grace_period_seconds, Prefect gives the deployment concurrency lease 5 minutes during provisioning before it can be considered lost.
If your run startup can take longer than that, set a higher value explicitly.
Example:
python
from prefect.client.schemas.objects import ConcurrencyOptions, ConcurrencyLimitStrategy
flow.from_source(...).deploy(
name="my-deployment",
work_pool_name="my-pool",
concurrency_limit=5,
concurrency_options=ConcurrencyOptions(
collision_strategy=ConcurrencyLimitStrategy.ENQUEUE,
grace_period_seconds=900,
),
)
If you want, I can also show where to set this in prefect.yaml.Robert Szomolanyi
06/18/2026, 10:15 AMprefect.exceptions.CancelledRun: Deployment concurrency slot lost during provisioning - no slots available to continue execution
But it does not take 5 minutes ...
Here is the log :
Worker 'ProcessWorker ee406fd7-8d70-4135-95c6-40ac4344a8d2' submitting flow run '019ed936-2ebc-730b-ba5f-dcc6be5836a4'
08:33:30 AM
prefect.flow_runs.worker
Starting flow run process...
08:33:30 AM
prefect.flow_runs.runner
Flow run 'watchful-barnacle' submitted to infrastructure
08:33:30 AM
prefect.flow_runs.worker
Running 1 deployment pull step(s)
08:33:31 AM
prefect.flow_runs
Executing deployment step: set_working_directory
08:33:31 AM
prefect.flow_runs
Deployment step 'set_working_directory' completed successfully
08:33:31 AM
prefect.flow_runs
All deployment steps completed successfully
08:33:31 AM
prefect.flow_runs
Changing working directory to '/apphome/clag'
08:33:31 AM
prefect.flow_runs
Importing flow code from 'scanner.main:simple_scanner'
08:33:31 AM
prefect.flow_runs
Engine execution exited with unexpected exception
Traceback (most recent call last):
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 2191, in run_flow
ret_val = run_flow_sync(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 2036, in run_flow_sync
return engine.state if return_type == "state" else engine.result()
^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 776, in result
_result = self.state.result(raise_on_failure=raise_on_failure, _sync=True) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/compatibility/async_dispatch.py", line 94, in wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/client/schemas/objects.py", line 394, in result
return run_coro_as_sync(
^^^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/utilities/asyncutils/__init__.py", line 207, in run_coro_as_sync
return call.result()
^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
return self.future.result(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
result = await coro
^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/utilities/asyncutils/__init__.py", line 188, in coroutine_wrapper
return await task
^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/states.py", line 85, in get_state_result
return await _get_state_result(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/states.py", line 157, in _get_state_result
raise await aget_state_exception(state)
prefect.exceptions.CancelledRun: Deployment concurrency slot lost during provisioning - no slots available to continue execution
08:33:32 AM
prefect.flow_runs
Process for flow run 'watchful-barnacle' exited with status code: 1; Process exited with a general error.
08:33:32 AM
prefect.flow_runs.runner
Check the flow run logs for an unhandled exception or assertion error.
08:33:32 AM
prefect.flow_runs.runner
Check the flow run logs for an unhandled exception or assertion error.Marvin
06/18/2026, 10:15 AMMarvin
06/18/2026, 10:17 AM300s default is the lease duration, but this error does not strictly mean “the full 300 seconds elapsed.” It means that when Prefect tried to move the run from Pending to Running, the concurrency lease was already gone and no slot was available to reacquire.
In your log, that check is happening almost immediately after import starts:
text
08:33:30 submitting flow run
08:33:30 Starting flow run process...
08:33:31 Importing flow code ...
08:33:31 Engine execution exited ...
08:33:32 CancelledRun: Deployment concurrency slot lost during provisioning
So this is not “startup took more than 5 minutes.”
It’s more likely one of these:
Most likely causes
1. Another competing run took the slot
- Your run had a slot while being provisioned
- By the time Prefect validated the Pending -> Running transition, that slot was no longer valid
- Another run for the same deployment may have acquired it
2. Worker/server version mismatch
- This is a known thing to check with deployment concurrency leasing
- If the worker is older and the server is newer, lease renewal/validation behavior can be inconsistent
3. A very small custom grace period is actually configured
- Even if you expected the default, the deployment may explicitly set a lower value
4. Concurrency lease/revocation race
- The run can lose the lease due to orchestration timing, even very quickly, if the server decides the original lease is no longer valid and cannot reacquire a slot
Important point
This message does not mean:
- your code import failed
- your flow function is broken
- Prefect waited 5 minutes
It means:
- Prefect cancelled the run before execution could continue, because deployment concurrency enforcement rejected the final transition to Running.
What I’d check first
1. Check Prefect versions on both worker and server/API
- especially whether the worker is older than the server
- I strongly recommend using a recent Prefect 3.x on both sides
2. Check deployment concurrency settings
- concurrency_limit
- concurrency_options
- grace_period_seconds
- collision strategy
3. Check whether multiple runs of the same deployment were starting together
- if limit is 1, this becomes much easier to hit
4. Switch collision strategy to ENQUEUE
- this is often safer than cancellation when there is contention
Likely fix
If this is a deployment with concurrency control, I’d try:
python
from prefect.client.schemas.objects import ConcurrencyOptions, ConcurrencyLimitStrategy
flow.from_source(...).deploy(
name="my-deployment",
work_pool_name="my-pool",
concurrency_limit=1,
concurrency_options=ConcurrencyOptions(
collision_strategy=ConcurrencyLimitStrategy.ENQUEUE,
grace_period_seconds=900,
),
)
That helps in two ways:
- longer lease during provisioning
- queue/wait behavior instead of canceling when slots are contested
What would help narrow it down
Can you share:
- your Prefect version on the worker
- your Prefect server/cloud version if self-hosted
- the deployment’s concurrency config
- whether concurrency_limit=1 for that deployment
- whether multiple runs were triggered around the same time
If you paste the deployment definition or prefect.yaml, I can tell you exactly which setting is causing this.Robert Szomolanyi
06/18/2026, 10:33 AMschedules=[
DeploymentScheduleCreate(
schedule=IntervalSchedule(interval=timedelta(minutes=5)),
active=True,
max_scheduled_runs=1,
)
],
concurrency_limit=1,
concurrency_options=ConcurrencyOptions(
collision_strategy=ConcurrencyLimitStrategy.CANCEL_NEW, # I don't want to plan next run, if old one is running
),
Only this run was triggered. Scheduling is set to every 5 minutes.Marvin
06/18/2026, 10:33 AMMarvin
06/18/2026, 10:33 AM3.7.1
- API 3.7.1
- background services 3.7.1
- concurrency_limit=1
- collision_strategy=CANCEL_NEW
- schedule every 5 minutes
- only one visible run triggered
this should not be a version mismatch issue.
What’s most likely happening is:
Your deployment concurrency limit is 1, and at the moment this run transitions to Running, Prefect thinks the only slot is already occupied.
With CANCEL_NEW, Prefect will cancel instead of queueing/retrying if it cannot keep or reacquire the slot.
That matches your error exactly.
Why this can happen even with “only one run”
There are a few realistic explanations:
1. A previous run still held the deployment slot briefly
- even if the prior run looked finished, cancellation/finalization/cleanup may not have fully released the slot yet
- with a 5-minute schedule and concurrency_limit=1, a boundary race is possible
2. The same run temporarily lost its lease and then failed to reacquire it
- at Pending -> Running, Prefect validates the deployment slot again
- if lease validation fails and slot count says 0 available, the run is cancelled
3. There may be another run attempt/reschedule not obvious from the UI at first glance
- especially around scheduled runs, retries, or a run that was created then rescheduled/cancelled quickly
4. CANCEL_NEW makes this behavior harsher
- even a transient slot conflict leads to cancellation instead of waiting
Most important insight
With:
python
concurrency_limit=1
collision_strategy=ConcurrencyLimitStrategy.CANCEL_NEW
you are effectively telling Prefect:
“If anything else appears to hold the deployment slot, cancel the new run immediately.”So even a short-lived slot accounting conflict can produce exactly what you saw. What I recommend Option 1: Switch to
ENQUEUE
This is the safest fix if you want to eliminate these transient cancellations.
python
concurrency_options=ConcurrencyOptions(
collision_strategy=ConcurrencyLimitStrategy.ENQUEUE,
)
That does not allow overlap.
It just means: “wait until the old run is really done” instead of canceling.
If your concern is “I don’t want stacked future runs,” ENQUEUE may still be acceptable if you also keep scheduling tight and limit what gets created.
Option 2: Keep CANCEL_NEW, but expect occasional cancellations
If you truly want “drop the new run if the old one is still in any way active,” then this behavior is consistent with that policy.
Option 3: Add a larger grace period
This may reduce lease-loss edge cases:
python
concurrency_options=ConcurrencyOptions(
collision_strategy=ConcurrencyLimitStrategy.CANCEL_NEW,
grace_period_seconds=900,
)
But I suspect your main issue is less about lease duration and more about CANCEL_NEW with concurrency_limit=1.
My read of your case
Because the failure happens within ~2 seconds, I do not think:
- the run exceeded the default 5-minute grace period
- worker/API version mismatch is involved
I do think:
- the deployment slot was not available at the exact orchestration check
- and CANCEL_NEW caused immediate cancellation rather than waiting
Suggested next step
I’d try this first:
python
concurrency_limit=1,
concurrency_options=ConcurrencyOptions(
collision_strategy=ConcurrencyLimitStrategy.ENQUEUE,
grace_period_seconds=300,
)
That still preserves no parallel runs, but avoids cancellation due to a tiny timing race.
If you really need “skip rather than wait,” then CANCEL_NEW is the right semantic choice — but then these occasional cancellations are an expected side effect when slot availability is ambiguous for even a moment.
One more thing to inspect
Check whether the previous scheduled run was still in a finalizing state around 08:33:30, even if it looked done. With a 5-minute interval and limit 1, that’s the most plausible trigger.Marvin
06/18/2026, 10:33 AMRobert Szomolanyi
06/18/2026, 10:42 AMMarvin
06/18/2026, 10:43 AMMarvin
06/18/2026, 10:43 AM8:23, 8:28, 8:33 is cancelled within 2–3 seconds with:
text
Deployment concurrency slot lost during provisioning - no slots available to continue execution
then this is probably not real contention from another active run.
With concurrency_limit=1, repeated immediate cancellations strongly suggest the deployment’s concurrency slot is effectively stuck as occupied or Prefect’s slot/lease state is inconsistent.
What’s likely happening
Prefect is doing something like:
1. schedule creates the run
2. worker starts provisioning it
3. when transitioning Pending -> Running, Prefect checks the deployment concurrency slot
4. Prefect believes no slot is available
5. because strategy is CANCEL_NEW, the run is cancelled immediately
If that happens for every run, then one of these is likely true:
1. A stale active slot is stuck on the deployment concurrency limit
- Prefect thinks one run is still holding the slot
- but that run is no longer actually running
2. A stale/orphaned lease exists
- the lease backing deployment concurrency was not cleaned up correctly
- new runs cannot reacquire the slot
3. There’s a server-side state inconsistency/bug around deployment concurrency
- especially since all runs fail the same way immediately
Because this is self-hosted, this is much more plausible than “another hidden run is always starting at the exact same time.”
Why the first cancelled run can poison the next ones
If one run loses the slot during provisioning and cleanup does not fully release the slot, then the next scheduled run sees:
- slot already occupied
- CANCEL_NEW => cancel
- cleanup again may not clear it
- repeat forever
That would explain the exact loop you’re seeing every 5 minutes.
What to check right now
Please inspect the deployment’s concurrency state in the UI/API and look for:
- active slot count = 1
- no actual running flow runs
If that’s true, you’ve basically confirmed a stuck concurrency slot.
Fastest test
The quickest way to verify this is the deployment concurrency state itself.
Also, a very useful test is:
temporarily remove deployment concurrency
Set:
python
concurrency_limit=None
or redeploy without concurrency_limit.
If the next run starts normally, then you’ve confirmed the issue is specifically the deployment concurrency slot state, not your flow code or worker.
Second test
Change only this:
python
collision_strategy=ConcurrencyLimitStrategy.ENQUEUE
If the run no longer gets immediately cancelled, that also confirms the failure is happening at the concurrency check rather than flow startup.
Most likely root cause in your case
Based on the repeated 5-minute failures, I think the most likely explanation is:
A stale deployment concurrency slot / orphaned lease is stuck after an earlier cancelled run, so every new scheduled run sees the slot as unavailable and gets cancelled by CANCEL_NEW.
What to do
Best immediate workaround
1. redeploy without deployment concurrency, or
2. switch to ENQUEUE
Best diagnostic step
- check whether the deployment concurrency slot count says occupied while no run is running
If you want skip-not-queue semantics
Deployment concurrency with CANCEL_NEW may not be the best fit for “run every 5 minutes, but silently skip if previous run is still active.” It can be brittle if slot accounting gets stuck.
A more robust pattern is often:
- allow scheduling
- at flow start, check whether a previous run is still active
- if yes, exit cleanly with Completed("skipped") or similar
That avoids dependency on deployment slot leasing for skip semantics.
Docs
- Deployments: Deployments
- Work pools/workers: Work pools and workers
If you want, I can help you with either:Marvin
06/18/2026, 10:43 AMCANCEL_NEW.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by