<@ULVA73B9P> my prefect deploy just started giving...
# ask-marvin
c
@Marvin my prefect deploy just started giving me this error:
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f2a31941910
there were no code changes or deploys. what does this error mean?
m
thought for 247.4 seconds
Short answer: that error means the run was intentionally interrupted. In Prefect 3.x we use AnyIO’s “cancel scopes”; when something cancels a run (manual action, a timeout, or the infrastructure terminating the process), the engine raises
asyncio.exceptions.CancelledError: Cancelled by cancel scope ...
. It’s not a bug in your code, it’s an interruption signal. Common reasons this suddenly appears without code changes: - Someone clicked “Cancel” in the UI or used the CLI to cancel the run. - A timeout fired:
@flow(timeout_seconds=...)
or
@task(timeout_seconds=...)
. - The underlying infrastructure terminated the process: - Kubernetes pod eviction/preemption/node scale-down, spot interruption, or job/controller cleanup - ECS task stopped - Worker shutdown/restart (received SIGTERM) - An Automation that cancels long/late runs was added or changed. How to confirm and diagnose: - In the Prefect UI, open the flow run: - Check the State History/timeline: you’ll usually see it go to “Cancelling” then “Cancelled”. The message often reads “Execution was cancelled by the runtime environment.” - Check the Logs around the transition. - Check your worker/infrastructure logs at the same timestamp for SIGTERM, eviction, preemption, or restarts. - Verify you (or teammates) didn’t add/change: - Flow/task timeouts in code - Automations that cancel late/long runs - Work pool/infra policies (e.g., K8s pod disruption, spot scaling) Helpful references: - Cancel workflows in Prefect 3.x - States and state transitions A couple clarifying questions so I can point you to the most likely cause: - Do you see the error in flow run logs, or when running the
prefect deploy
CLI itself? - What work pool/worker type are you using (Kubernetes, ECS, Docker, process)? - Do you have any flow/task timeouts or Automations configured? - Is this hitting all runs or just specific ones, and when did it start? If you share a flow run ID, your work pool type, and a snippet of worker logs around the time of cancellation, I can help pinpoint the trigger.
c
@Marvin ah, I actually see this error above:
Copy code
pydantic_core._pydantic_core.ValidationError: 1 validation error for DeploymentResponse
work_queue_id
  UUID input should be a string, bytes or UUID object [type=uuid_type, input_value=None, input_type=NoneType]
m
thought for 236.7 seconds
Got it — that ValidationError is the real root cause. The
CancelledError
you saw is just a follow‑on effect from the async job being torn down after the failure. What the error means -
ValidationError for DeploymentResponse
with
work_queue_id
being
None
indicates your client is parsing a Deployment response that doesn’t match the schema it expects. In Prefect 3.x, deployments are tied to a work pool (via
work_pool_name
), not a
work_queue_id
. - This usually happens when there’s a version mismatch between your Prefect client and the API you’re talking to (e.g., a 3.x client pointed at a 2.x server, or vice versa), or when a 2.x-style deployment/response includes fields your client doesn’t expect. How to fix 1) Confirm versions and target - Run
prefect version
to see your installed package version. - Check where you’re connected (Cloud vs OSS server) and ensure the API/server major version matches your client. For self‑hosted OSS, major versions must match. - If your client is 3.x but you’re pointing to a 2.x server, either: - Upgrade the server to 3.x, or - Use a 2.x client:
pip install "prefect<3"
- If both are 3.x, make sure your deployments use work pools, not work queues. 2) If you’re on Prefect 3.x, deploy with a work pool (not a work queue) - Update deployment code to use `work_pool_name`:
Copy code
from prefect import flow

flow_obj = flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow.py:my_flow",
)

flow_obj.deploy(
    name="my-deployment",
    work_pool_name="your-work-pool",
)
- Make sure the work pool exists and a worker is polling it. - Docs: Manage work pools 3) If you’re intentionally on Prefect 2.x everywhere - Ensure your deployment references a valid work queue and that the queue exists. A missing/removed queue can surface as
work_queue_id=None
in responses. Why the cancel showed up - After the schema validation error, the async runtime cancels pending tasks, which surfaces as
asyncio.exceptions.CancelledError
. It’s a symptom, not the root cause. If you paste: - The output of
prefect version
- Whether you’re using Cloud or OSS (and OSS server version if applicable) - How you’re creating the deployment (CLI or code snippet) …I can give you the exact next command(s) to run. Helpful refs: - Prefect Server and version compatibility - Work pools and deployments in Prefect 3.x
c
@Marvin my poetry version output is:
Copy code
Version:             3.1.1
API version:         0.8.4
Python version:      3.12.7
Git commit:          6b50a2b9
Built:               Fri, Nov 8, 2024 12:38 PM
OS/Arch:             linux/x86_64
Profile:             ephemeral
Server type:         cloud
Pydantic version:    2.10.0b1
My deployment is not using any work pools. I tried adding
work_pool_name
to my deploy but got error
TypeError: Flow.serve() got an unexpected keyword argument 'work_pool_name'
m
thought for 86.6 seconds
Thanks — that version output helps. You’re on Prefect 3.1.1 (Cloud), so deployments should be associated with a work pool. Two key points: 1) The ValidationError about
DeploymentResponse.work_queue_id
means something in your path is still using an old 2.x “work queue” shape. In 3.x, deployments target a work pool (via
work_pool_name
), not a
work_queue_id
. That error commonly shows up when: - A 2.x worker is still polling (it expects
work_queue_id
) - A 2.x-style deploy config or code is being used - A 3.x deploy attempts to parse a response that still includes/assumes old queue fields 2) The
TypeError: Flow.serve() got an unexpected keyword argument 'work_pool_name'
happened because
.serve()
doesn’t take
work_pool_name
. In 3.x,
.serve()
is for local/dev serving;
.deploy()
is how you create a Cloud deployment tied to a work pool. What to do - Ensure you’re actually using
.deploy(...)
(not
.serve(...)
) to create the deployment and that you specify a work pool:
Copy code
from prefect import flow

my_flow = flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow_module.py:my_flow",
)

my_flow.deploy(
    name="my-deployment",
    work_pool_name="my-work-pool",
)
- Make sure a worker is running against that pool (and that it’s Prefect 3.x). If you have any old 2.x workers still polling, stop/upgrade them. - If you were using a 2.x-style configuration (e.g., anything referencing “work queue”), update it to 3.x with a work pool. - If you were relying on
.serve()
, remove
work_pool_name
there—`.serve()` doesn’t create a Deployment in Cloud. Use
.deploy()
for Cloud deployments. Helpful docs: - Work pools and workers - Deploy flows from source in Prefect 3.x To pinpoint the mismatch, could you share: - How you’re creating the deployment right now (CLI or code snippet)? - The worker type and version that’s polling your runs (Kubernetes/ECS/Docker/process)? - Whether you have any older 2.x workers still running?