https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> we're on `"prefect==3.4.7"` and `pref...
# ask-marvin
j
@Marvin we're on 
"prefect==3.4.7"
and 
prefecthq/prefect-aws:0.7.7-python3.12-prefect3.6.29
for our ecs worker. there are two issues with prefect ecs we are unsure how to resolve: 1. flow runs that are manually cancelled get stuck in a cancelling state 2. the concurrency limit for a given deployment's state gets out of sync. it claims that slots are taken despite no flow runs attached both our are big deal to us, especially 2, because our throughput takes a hit. what should we do to resolve this?
m
✅ thought for 109.3 seconds
Given the versions you shared, my strong recommendation is: upgrade both Prefect and your ECS worker image before doing anything else. You’re on: - 
prefect==3.4.7
- ECS worker image 
prefecthq/prefect-aws:0.7.7-python3.12-prefect3.6.29
That version mix is old enough that both symptoms line up with known fixes made later in Prefect 3.x. What’s likely happening 1. Flow runs stuck in 
Cancelling
 - Older Prefect versions could leave runs in 
CANCELLING
if the worker/infrastructure teardown path did not complete cleanly. - A cleanup mechanism for stuck cancelling runs was added later in Prefect. - Relevant release note: Prefect 3.7.5 release notes - Specifically mentions: 
Add cancelling timeout cleanup producer
in #22238 2. Deployment concurrency slots getting “stuck” - There were fixes for deployment concurrency lease handling in later 3.x releases. - This is consistent with a known class of bugs where the lease/slot was acquired but not reliably released during state transitions or teardown. - One relevant fix was “Preserve deployment concurrency lease ID across state transitions” in #21280 3. ECS-specific worker/event handling - There was also a later 
prefect-aws
ECS fix around observer ack timing, which could contribute to state drift / missed cleanup behavior: - #21927 What I would do 1. Upgrade to a newer, aligned version pair Don’t keep running: - Python package: 
prefect==3.4.7
- Worker image built for: 
prefect3.6.29
That version skew is not ideal by itself. I’d move to a matched and current-enough pair, ideally at least: - 
prefect >= 3.7.5
- a matching 
prefect-aws
ECS worker image from the same era or newer For example, use a worker image tagged with the same Prefect core version you install in your environment. You can review current releases here: - Core Prefect releases: Prefect releases - Prefect AWS releases: prefect-aws releases 2. Make sure your worker image and control-plane package are aligned Right now you have: - app/runtime package: 
3.4.7
- worker image runtime: 
3.6.29
Even if this is not the sole cause, it increases the chance of weird orchestration/state behavior. I would standardize on one version. 3. After upgrade, retest both behaviors specifically Create two simple tests: - a long-running flow that you manually cancel - a deployment with a low concurrency limit, then run/cancel/complete repeatedly and verify slots are released 4. If you cannot upgrade immediately, use operational mitigations Short term, you may need manual cleanup for both symptoms. For stuck 
CANCELLING
runs, you can force a terminal state via the API:
Copy code
python
from prefect.client.orchestration import get_client
from prefect.states import Cancelled
import asyncio

async def force_cancel(flow_run_id: str):
    async with get_client() as client:
        await client.set_flow_run_state(
            flow_run_id=flow_run_id,
            state=Cancelled(message="Manual recovery from stuck CANCELLING state"),
        )

asyncio.run(force_cancel("YOUR_FLOW_RUN_ID"))
I have not verified `set_flow_run_state`’s exact client signature in your installed version, so if you want, I can help you with the exact snippet for 
3.4.7
. 5. Investigate whether the slots are truly orphaned vs hidden by filters Before assuming only UI drift, confirm via API/DB whether: - the deployment/global concurrency limit shows occupied slots - there are no corresponding active flow runs in 
RUNNING
, 
PENDING
, 
SCHEDULED
, 
CANCELLING
, etc. If you’re self-hosting the API/database, DB inspection may help. If you’re on Cloud, use the API/UI. Bottom line
For your case, I would treat this as a version/bugfix issue first, not a configuration problem. Recommended next move: - upgrade to at least 
Prefect 3.7.5
- upgrade 
prefect-aws
to a matching newer release - ensure the ECS worker image version matches the 
prefect
version used by your deployments/workers Why I’m confident that’s the right direction Your two symptoms map closely to known fixes: - stuck cancellations cleanup: #22238 - deployment concurrency lease preservation: #21280 - ECS observer ack fix: #21927 If you want, I can next help you with one of these: 1. a safe upgrade target/version matrix for 
prefect
+ 
prefect-aws
2. a runbook to clean up stuck concurrency slots today 3. a minimal repro/checklist to verify whether ECS cancellation is failing at the worker, task, or orchestration layer
j
i bumped prefect and prefect-aws. 1. the state transitions to cancelled in the UI, 2. but the last task in the flow run is still blue, executing 3. the ecs task is still running
@Marvin what's up w that?
m
✅ thought for 11.3 seconds
Fair question — the short version is: your versions are mismatched, and both are old enough to include bugs that were later fixed. Specifically: - your client/server/runtime is 
prefect==3.4.7
- your ECS worker image is 
prefecthq/prefect-aws:0.7.7-python3.12-prefect3.6.29
So your worker is running Prefect 3.6.29, while you say “we’re on” 3.4.7. That means at least part of your system is not on the same core version. Why that matters Prefect orchestration, worker behavior, cancellation handling, and concurrency lease cleanup all depend on tightly coordinated state transitions. When you combine: - older orchestration behavior - ECS-specific worker code - version skew between worker/runtime pieces …you increase the chance of exactly the kind of drift you’re seeing: - runs stuck in 
CANCELLING
- concurrency slots that never get released More concretely There are later fixes for: - stuck 
CANCELLING
cleanup: #22238 - concurrency lease preservation: #21280 - ECS observer ack behavior: #21927 So “what’s up with that?” = you’re very likely running into already-fixed bugs, made harder to reason about by version mismatch. One more important nuance I don’t want to overstate “mismatch” as the sole cause without confirming your architecture. There are a few possibilities: - Case 1: self-hosted server/API is 3.4.7, ECS worker image is 3.6.29 - this is definitely skew, and I’d fix it first - Case 2: your flow code environment is 3.4.7, but workers are 3.6.29, and server is something else - still skew, still suspicious - Case 3: you’re on Prefect Cloud, but your runtime/worker pieces are split between 3.4.7 and 3.6.29 - less about server mismatch, more about worker/runtime behavior still being inconsistent and outdated Best next step Can you clarify which of these 
3.4.7
refers to? 1. your application environment running flow code 2. your self-hosted Prefect server/API 3. both And are you on: - Prefect Cloud, or - self-hosted Prefect server? That will let me tell you whether this is primarily: - a worker bug - an API/server orchestration bug - or a cross-version compatibility problem.
j
nice catch, yeah why is my worker's version behind what i've declared in the image? worker says: 
3.6.29
image defines: 
prefecthq/prefect-aws:0.7.9-python3.12-prefect3.7.5
@Marvin
m
✅ thought for 3.4 seconds
Looks like your last message came through empty. Can you send: - whether you’re on Prefect Cloud or self-hosted server - what 
prefect==3.4.7
refers to: - flow code environment - server/API - both If you want, paste:
Copy code
bash
prefect version
and, if self-hosted, the server version too. That’ll let me give you a precise answer instead of guessing.
j
@Marvin we're using prefect cloud. worker says: 
3.6.29
image defines: 
prefecthq/prefect-aws:0.7.9-python3.12-prefect3.7.5
m
✅ thought for 12.1 seconds
Ah — that changes the picture a lot. If you're on Prefect Cloud, then the server-side version is not your issue here. And if: - your worker reports 
3.6.29
- but your image tag says 
prefecthq/prefect-aws:0.7.9-python3.12-prefect3.7.5
then the main thing “up with that” is: your running worker does not appear to be using the image/version you think it is. That is the first thing I would investigate. Why this matters You previously mentioned: - 
prefect==3.4.7
- worker image around 
0.7.7 ... prefect3.6.29
Now you’re saying: - Cloud - worker runtime says 
3.6.29
- configured image says 
0.7.9 ... prefect3.7.5
Those two facts are inconsistent. If the image really is 
...prefect3.7.5
, I would expect 
prefect version
inside that running worker container to report 
3.7.5
, not 
3.6.29
. So likely one of these is true: 1. old ECS task definition / old image revision is still being used 2. worker service was not redeployed after changing the image 3. there are multiple workers and the one picking up work is still old 4. the container is overriding/installing a different 
prefect
version at startup 5. you’re checking version in a different container than the one actually running the worker What I’d do first 1. Verify the actual running image digest/tag in ECS In AWS ECS, check the running task for the worker service and confirm: - task definition revision - container image URI/tag - image digest actually pulled Do not rely only on what is “configured in code”; verify the running task. 2. Exec into the running worker container and check:
Copy code
bash
prefect version
python -c "import prefect; print(prefect.__version__)"
pip show prefect
pip show prefect-aws
If those still say 
3.6.29
, then the 
3.7.5
image is not actually what’s running, or something is downgrading Prefect in-container. 3. Check for startup-time installs Look for any of these in your Dockerfile / entrypoint / ECS command: - 
pip install prefect==...
- 
uv pip install ...
- 
poetry install
- app startup scripts that reinstall dependencies A common pattern is: - base image has Prefect 3.7.5 - app layer installs 
prefect==3.6.29
from 
requirements.txt
- final running container reports 3.6.29 4. Check whether multiple workers are registered If one old ECS worker is still alive, it may be the one processing cancellations and acquiring concurrency slots. You can inspect workers in Cloud UI and verify: - worker names - last polled time - work pool association - whether old workers are still online Why this probably explains your symptoms If the live worker is really still 
3.6.29
, then: - you may not actually have the ECS observer fix you thought you deployed - you may not have the cancellation improvements you thought you deployed - you may still be seeing the older concurrency behavior So before going deeper into Prefect behavior, I would resolve the version discrepancy. What I think is most likely Most likely: - the ECS worker service is still running an old task revision or image - or your app layer/install step overwrote the Prefect version inside the image Fastest way to prove it From inside the running worker container, gather:
Copy code
bash
prefect version
python -c "import prefect, prefect_aws; print(prefect.__version__); print(prefect_aws.__version__)"
pip freeze | grep prefect
If you paste that output, I can tell you immediately whether: - the wrong image is running - the image is being mutated at runtime - or there’s something else unusual going on Bottom line Right now I would not assume the 
0.7.9 / prefect3.7.5
image is actually deployed. The mismatch between: - reported worker version 
3.6.29
- expected image version 
3.7.5
is the biggest red flag in the whole thread.
j
this is a bug w my cicd flow
@Marvin how does cancellation work? is it tied to whatever task is live?
m
✅ thought for 122.8 seconds
Yes — cancellation is tied to a live worker/infrastructure observer, not directly to “whatever task is live” in the sense of Python task execution. For Prefect Cloud + ECS worker, the rough model is: 1. You click Cancel in UI / call cancel via API 2. Prefect changes the flow run state to 
CANCELLING
 3. A worker notices that state change 4. The worker tries to stop the backing infrastructure - for ECS, that means calling ECS to stop the task 5. Once cleanup succeeds, the flow run moves to 
CANCELLED
 So 
CANCELLING
is an in-between state, not the final one. Important nuance for ECS The cancellation is tied to: - a worker being alive and observing the run, and - the run having a valid infrastructure identifier for the ECS task In practice, the worker needs to know: “this flow run corresponds to ECS task X in cluster Y”, then it can stop that ECS task. What it is not tied to It is not about whichever Prefect task function happens to be currently executing inside your flow. Prefect isn’t interrupting a specific Python task object directly. Instead, it cancels by stopping the execution environment backing the flow run. For ECS that usually means: - stopping the whole ECS task/container running the flow run Why runs can get stuck in 
CANCELLING
 That happens if: - the worker never sees/processes the cancellation - the worker crashes - the ECS task can’t be stopped cleanly - the infrastructure PID/task identifier is missing or wrong - the worker version has a bug in cancellation handling That’s why your version mismatch matters so much: - if the worker that is actually live is older than you expect, it may be missing fixes in this path One more nuance There is also a newer cleanup mechanism in later Prefect versions so runs don’t remain in 
CANCELLING
forever: - #22238 So in newer versions: - ideal path: worker sees cancel, stops ECS task, marks 
CANCELLED
- fallback path: cleanup service eventually forces the run to 
CANCELLED
For your exact question: “is it tied to whatever task is live?” Best concise answer: - No, not to an individual Prefect task run - Yes, to the live worker + live ECS infrastructure record for that flow run If you want, I can next explain: 1. how deployment concurrency slots are supposed to be released during cancellation, or 2. exactly what to inspect in Cloud/ECS when a run stays in 
CANCELLING
j
@Marvin ok, i have my worker and deployment prefect versions both on 
3.7.5
. 1. the UI shows that the flow run is cancelled 2. the ecs task stopped 3. BUT, there is still an active task in the flow run, the one that was live when the flow run was cancelled 3 seems like a bug?
m
✅ thought for 251.9 seconds
Yes — if it stays active for more than a short cleanup window, that sounds like a bug or at least a cleanup failure. But there’s an important nuance: In Prefect today, flow-run cancellation and task-run cancellation are not necessarily synchronous. What can happen is: 1. flow run is marked 
CANCELLED
 2. ECS task is stopped 3. task runs that were in-flight do not get updated instantly 4. a background cleanup process later marks those child task runs terminal So a brief period where: - flow run = 
CANCELLED
- one task run still looks active can be expected. What is not expected If that task remains active for a while — especially minutes, not seconds — then yes, I would treat that as a bug / cleanup failure. From the code path, child task-run cleanup is handled asynchronously by server-side cancellation cleanup logic, not inline with the flow run transition. So there is a race/window. Rule of thumb - A few seconds / short delay: probably expected - Persistent active task after the flow is cancelled and ECS task is gone: not good, likely a bug Why this happens conceptually When the ECS task is killed, the Python process running the task disappears immediately. That means the task never gets a chance to report: - 
Cancelled
- 
Failed
- 
Crashed
So Prefect has to reconcile that afterward from the outside. If the reconciliation succeeds, the task run should eventually be marked terminal. If it doesn’t, you’re left with an orphaned “active” task run. What I’d check next Please confirm: - how long the task stays active after the flow run is shown as 
CANCELLED
- whether it is shown as 
RUNNING
, 
PENDING
, or something else - whether this stale task run is also what’s causing your concurrency slot issue Those details matter. Most likely answer to your question “3 seems like a bug?” - Yes, if it persists - Not necessarily, if it’s only transient What you can do right now If you’re self-hosting, I’d suggest checking cleanup service settings — but since you’re on Prefect Cloud, that part is managed. So for Cloud, if you can reproduce this reliably on 
3.7.5
with: - flow run cancelled - ECS task stopped - child task run remains active for a long time then that’s strong evidence of a product bug. Useful thing to capture for a repro If you want, I can help you write a minimal repro flow, but the ideal data to capture is: - flow run ID - task run ID - timestamps for: - task start - cancel click - flow run became 
CANCELLED
- ECS task stopped - task run still active at 
T+N minutes
Since I can’t file a ticket for you, the best public route if you want Prefect engineering eyes is: - GitHub issues: prefect issues - or contact Prefect: prefect contact If you want, I can help you turn this into a tight minimal repro + issue draft so you can post it cleanly.
j
@Marvin file a ticket for me
m
✅ thought for 9.9 seconds
I can’t file it on your behalf, but I can give you a ready-to-paste issue. Use: Open a Prefect issue Paste this:
Copy code
text
Title:
Cancelled flow run on ECS leaves in-flight task run active after ECS task has stopped

Body:

## Summary

On Prefect Cloud with ECS workers, cancelling a flow run can result in this sequence:

1. Flow run transitions to `CANCELLED`
2. ECS task is stopped
3. The task run that was active at cancellation time remains active/non-terminal in the UI/API

There may be a brief reconciliation window where this is expected, but in our case the task run appears to remain active after the flow run is already `CANCELLED` and the ECS task is gone.

## Environment

- Prefect Cloud
- Worker image: `prefecthq/prefect-aws:0.7.9-python3.12-prefect3.7.5`
- Worker/runtime Prefect version: `3.7.5`
- ECS worker

## Observed behavior

After manually cancelling a flow run:

- the flow run is shown as `CANCELLED`
- the ECS task stops successfully
- the in-flight task run remains active/non-terminal instead of being reconciled to a terminal state

## Expected behavior

Once the flow run is `CANCELLED` and the backing ECS task has stopped, child task runs that were in-flight should be reconciled to a terminal state within a short period of time. They should not remain active indefinitely.

## Questions

- Is this expected to be eventually consistent only, or is this a bug?
- If expected to reconcile asynchronously, what is the expected upper bound for reconciliation?
- Could this leave deployment/task concurrency accounting out of sync?

## Reproduction sketch

A minimal repro seems to be:

1. Deploy a flow to ECS
2. Start a flow run with a long-running task
3. While the task is running, manually cancel the flow run
4. Observe:
   - flow run becomes `CANCELLED`
   - ECS task stops
   - task run that was active at time of cancellation remains active/non-terminal

## Example flow

```python
from prefect import flow, task
import time

@task
def long_task():
    time.sleep(600)

@flow
def cancel_test():
    long_task()

if __name__ == "__main__":
    cancel_test()
## Additional details to include Please let me know if it would help to provide: - flow run ID - task run ID - timestamps for cancellation and ECS task stop - screenshots/API responses showing the task run still active ``` What I recommend adding before submitting If you have them, add: - how long the task stayed active after cancellation - the exact task-run state shown in UI/API (
RUNNING
, 
PENDING
, etc.) - whether this also correlates with stuck concurrency slots If you want, I can also make this shorter and more engineering-focused before you submit it.
6 Views
Open in Slack
PreviousNext
Powered by