chris
11/13/2025, 6:01 PMasyncio.exceptions.CancelledError: Cancelled by cancel scope 7f2a31941910 there were no code changes or deploys. what does this error mean?Marvin
11/13/2025, 6:01 PMMarvin
11/13/2025, 6:06 PMasyncio.exceptions.CancelledError: Cancelled by cancel scope .... It’s not a bug in your code, it’s an interruption signal.
Common reasons this suddenly appears without code changes:
- Someone clicked “Cancel” in the UI or used the CLI to cancel the run.
- A timeout fired: @flow(timeout_seconds=...) or @task(timeout_seconds=...).
- The underlying infrastructure terminated the process:
- Kubernetes pod eviction/preemption/node scale-down, spot interruption, or job/controller cleanup
- ECS task stopped
- Worker shutdown/restart (received SIGTERM)
- An Automation that cancels long/late runs was added or changed.
How to confirm and diagnose:
- In the Prefect UI, open the flow run:
- Check the State History/timeline: you’ll usually see it go to “Cancelling” then “Cancelled”. The message often reads “Execution was cancelled by the runtime environment.”
- Check the Logs around the transition.
- Check your worker/infrastructure logs at the same timestamp for SIGTERM, eviction, preemption, or restarts.
- Verify you (or teammates) didn’t add/change:
- Flow/task timeouts in code
- Automations that cancel late/long runs
- Work pool/infra policies (e.g., K8s pod disruption, spot scaling)
Helpful references:
- Cancel workflows in Prefect 3.x
- States and state transitions
A couple clarifying questions so I can point you to the most likely cause:
- Do you see the error in flow run logs, or when running the prefect deploy CLI itself?
- What work pool/worker type are you using (Kubernetes, ECS, Docker, process)?
- Do you have any flow/task timeouts or Automations configured?
- Is this hitting all runs or just specific ones, and when did it start?
If you share a flow run ID, your work pool type, and a snippet of worker logs around the time of cancellation, I can help pinpoint the trigger.chris
11/13/2025, 6:08 PMpydantic_core._pydantic_core.ValidationError: 1 validation error for DeploymentResponse
work_queue_id
UUID input should be a string, bytes or UUID object [type=uuid_type, input_value=None, input_type=NoneType]Marvin
11/13/2025, 6:09 PMMarvin
11/13/2025, 6:13 PMCancelledError you saw is just a follow‑on effect from the async job being torn down after the failure.
What the error means
- ValidationError for DeploymentResponse with work_queue_id being None indicates your client is parsing a Deployment response that doesn’t match the schema it expects. In Prefect 3.x, deployments are tied to a work pool (via work_pool_name), not a work_queue_id.
- This usually happens when there’s a version mismatch between your Prefect client and the API you’re talking to (e.g., a 3.x client pointed at a 2.x server, or vice versa), or when a 2.x-style deployment/response includes fields your client doesn’t expect.
How to fix
1) Confirm versions and target
- Run prefect version to see your installed package version.
- Check where you’re connected (Cloud vs OSS server) and ensure the API/server major version matches your client. For self‑hosted OSS, major versions must match.
- If your client is 3.x but you’re pointing to a 2.x server, either:
- Upgrade the server to 3.x, or
- Use a 2.x client: pip install "prefect<3"
- If both are 3.x, make sure your deployments use work pools, not work queues.
2) If you’re on Prefect 3.x, deploy with a work pool (not a work queue)
- Update deployment code to use `work_pool_name`:
from prefect import flow
flow_obj = flow.from_source(
source="<https://github.com/your-org/your-repo>",
entrypoint="path/to/flow.py:my_flow",
)
flow_obj.deploy(
name="my-deployment",
work_pool_name="your-work-pool",
)
- Make sure the work pool exists and a worker is polling it.
- Docs: Manage work pools
3) If you’re intentionally on Prefect 2.x everywhere
- Ensure your deployment references a valid work queue and that the queue exists. A missing/removed queue can surface as work_queue_id=None in responses.
Why the cancel showed up
- After the schema validation error, the async runtime cancels pending tasks, which surfaces as asyncio.exceptions.CancelledError. It’s a symptom, not the root cause.
If you paste:
- The output of prefect version
- Whether you’re using Cloud or OSS (and OSS server version if applicable)
- How you’re creating the deployment (CLI or code snippet)
…I can give you the exact next command(s) to run. Helpful refs:
- Prefect Server and version compatibility
- Work pools and deployments in Prefect 3.xchris
11/13/2025, 6:23 PMVersion: 3.1.1
API version: 0.8.4
Python version: 3.12.7
Git commit: 6b50a2b9
Built: Fri, Nov 8, 2024 12:38 PM
OS/Arch: linux/x86_64
Profile: ephemeral
Server type: cloud
Pydantic version: 2.10.0b1
My deployment is not using any work pools. I tried adding work_pool_name to my deploy but got error TypeError: Flow.serve() got an unexpected keyword argument 'work_pool_name'Marvin
11/13/2025, 6:23 PMMarvin
11/13/2025, 6:24 PMDeploymentResponse.work_queue_id means something in your path is still using an old 2.x “work queue” shape. In 3.x, deployments target a work pool (via work_pool_name), not a work_queue_id. That error commonly shows up when:
- A 2.x worker is still polling (it expects work_queue_id)
- A 2.x-style deploy config or code is being used
- A 3.x deploy attempts to parse a response that still includes/assumes old queue fields
2) The TypeError: Flow.serve() got an unexpected keyword argument 'work_pool_name' happened because .serve() doesn’t take work_pool_name. In 3.x, .serve() is for local/dev serving; .deploy() is how you create a Cloud deployment tied to a work pool.
What to do
- Ensure you’re actually using .deploy(...) (not .serve(...)) to create the deployment and that you specify a work pool:
from prefect import flow
my_flow = flow.from_source(
source="<https://github.com/your-org/your-repo>",
entrypoint="path/to/flow_module.py:my_flow",
)
my_flow.deploy(
name="my-deployment",
work_pool_name="my-work-pool",
)
- Make sure a worker is running against that pool (and that it’s Prefect 3.x). If you have any old 2.x workers still polling, stop/upgrade them.
- If you were using a 2.x-style configuration (e.g., anything referencing “work queue”), update it to 3.x with a work pool.
- If you were relying on .serve(), remove work_pool_name there—`.serve()` doesn’t create a Deployment in Cloud. Use .deploy() for Cloud deployments.
Helpful docs:
- Work pools and workers
- Deploy flows from source in Prefect 3.x
To pinpoint the mismatch, could you share:
- How you’re creating the deployment right now (CLI or code snippet)?
- The worker type and version that’s polling your runs (Kubernetes/ECS/Docker/process)?
- Whether you have any older 2.x workers still running?