This is likely a known regression in 3.7.0 affecting ECS work pools. What’s happening A bug in Prefect 3.7.0 could cause ECS-submitted flow runs to get stuck in

Submitting

because ECS observer events were acknowledged too early, so the state transition event could be lost. Recommended fix Upgrade to 3.7.1 or later for both

prefect

and

prefect-aws

, then restart your worker(s).

bash
pip install -U prefect prefect-aws

Relevant links: - Prefect 3.7.1 release notes - Fix: Ensure ECS observer ack waits for handlers What to do next 1. Upgrade

prefect

and

prefect-aws

2. Restart the ECS worker / worker service 3. Re-run one affected deployment 4. Check whether newly submitted runs move past

Submitting

Helpful checks - Inspect worker logs in CloudWatch for ECS observer / SQS processing errors - Confirm the ECS tasks are actually being created in AWS - Check whether only runs created on 3.7.0 are affected If you want, I can also help you with a step-by-step ECS-specific verification checklist (worker version, pool config, logs, and AWS-side checks).

Short answer: no — the new infrastructure decorators in 3.7.x should not require any backward-compatibility setting changes for existing ECS work pools. They were promoted to GA, but old usage is still intended to keep working. So if your runs are still stuck in

Submitting

, I would look elsewhere. What did change If you were importing decorators directly, the recommended import path changed:

python
# older / experimental path
from prefect_aws.experimental.decorators import ecs

# recommended in 3.7+
from prefect_aws.decorators import ecs

That said, this is not what would usually cause runs to sit in

Submitting

in an ECS work pool. What is more likely For ECS work pools,

Submitting

usually means the worker accepted the run but is failing before Prefect sees successful infrastructure submission / state transition. The most likely things to verify are: 1. Worker version alignment Make sure both the submitting environment and the worker environment are on compatible versions, especially

prefect

and

prefect-aws

. On the worker container:

bash
   prefect version
   pip show prefect prefect-aws

2. Worker logs Check the ECS worker logs in CloudWatch for: - ECS API permission errors - task definition registration errors - S3/code bundle/storage errors - worker channel / work pool sync errors 3. Work pool config Inspect the work pool and verify the base job template still looks valid:

bash
   prefect work-pool inspect <pool-name>

Things to confirm: - cluster name - launch type - task execution role ARN - task role ARN - network config / subnets / security groups - image value or job variable overrides 4. Code storage / source retrieval In Prefect 3.x, a deployment may depend on source retrieval or bundle/storage configuration. If the worker cannot retrieve the code package, the run can stall around submission/bootstrapping. 5. AWS-side ECS checks Look in ECS itself: - Are tasks being created at all? - Do they go immediately to

STOPPED

? - Is there an event message on the ECS service / task saying why? Important point about 3.7.x The main 3.7.x ECS-related bug was fixed after 3.7.0, but if you still see the issue after upgrading, that strongly suggests one of: - worker image still running old packages - stale worker task/service not restarted - work pool/job template issue - IAM / networking / ECS task startup problem - code retrieval/storage issue Useful links - Prefect 3.7.0 release notes - Prefect 3.7.1 release notes - ECS observer fix PR Best next step Can you share these 3 things? 1. The output of

prefect version

from: - the environment where you deploy - the ECS worker container 2. The output of:

bash
   prefect work-pool inspect <pool-name>

3. A snippet of the worker logs around one run that gets stuck in

Submitting

With those, I can help narrow down whether this is: - version mismatch - work pool template problem - AWS permission/network problem - code storage/source retrieval failure

That warning is highly relevant. Short version: your ECS worker is missing the ECS observer SQS queue, and in current Prefect 3.x ECS behavior that can absolutely explain runs getting stuck in

Submitting

. The important part is this:

text
SQS queue 'prefect-ecs-tasks-events' does not exist in region 'default'

This means the worker is trying to use the ECS observer, but it cannot find the SQS queue it expects for ECS task state events. What this affects That queue is used for ECS task event observation, including: - task state replication back into Prefect - crash detection - infrastructure state updates So even if ECS tasks are being submitted successfully, Prefect may not receive the event it needs to move the flow run out of

Submitting

. Also important:

"region 'default'"

usually means you did not explicitly set the observer queue region, so boto/AWS SDK is falling back to its default region resolution. That can also mean the worker is checking the wrong region. What to do 1. Deploy the ECS events infrastructure The warning itself points to the right fix: use

prefect-aws ecs-worker deploy-events

2. Set the observer env vars on the worker At minimum verify/set: -

PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_SQS_QUEUE_NAME

-

PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_SQS_QUEUE_REGION

Example:

text
   PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_SQS_QUEUE_NAME=prefect-ecs-tasks-events
   PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_SQS_QUEUE_REGION=us-east-1

3. Restart the worker after updating env/config If this is an ECS service, force a new deployment so the worker task picks up the changes. 4. Verify the queue exists in the same region the worker is using For example:

bash
   aws sqs get-queue-url --queue-name prefect-ecs-tasks-events --region us-east-1

What I would check right now - Is the SQS queue actually created? - Is it in the correct AWS region? - Does the worker task have permission to read from it? - Does the worker have the queue name configured correctly? - Are ECS task state change events being routed to that queue? Why I don’t think this is an infrastructure-decorator compatibility problem This points much more directly to missing ECS observer event infrastructure than to anything related to the new decorators. The decorators are not usually what controls whether ECS event replication works. Next best step Please share: - your AWS region - whether you ever ran

prefect-aws ecs-worker deploy-events

- the worker task env vars for the two observer settings above If you want, I can help you build the exact

prefect-aws ecs-worker deploy-events

command for your setup.