Great question — you’re on the right track thinking about a graceful cutover. In Prefect 3.x, there isn’t a built-in “drain” command for workers, and if you terminate a Process worker container while it’s running a flow, that flow process will be killed and the run will typically be marked as crashed (it’ll retry if your deployment has retries). To avoid stopping a worker that’s doing work, use one of these rollout patterns: Option A: Simple “drain” with pause/resume (least moving parts) - Pause the pool or queue(s) to prevent new work from starting:

prefect work-pool pause "<your-pool>"
  # OR
  prefect work-queue pause "<your-queue>" -p "<your-pool>"

- Wait for all running flow runs to complete. Use the UI or:

# Poll until no RUNNING flow runs remain
  while prefect flow-run ls --state RUNNING --limit 1 | grep -q RUNNING; do
    sleep 30
  done

- Stop the old worker containers (deactivate the old ACA revision). - Start the new worker containers (new image) and wait until they’re healthy. - Resume the pool or queue(s):

prefect work-pool resume "<your-pool>"
  # OR
  prefect work-queue resume "<your-queue>" -p "<your-pool>"

Notes: - Instead of pause, you can also set the pool concurrency to 0 to “freeze” new starts while letting current runs finish:

prefect work-pool set-concurrency-limit "<your-pool>" 0
  # later, raise it back to your normal limit

This has a similar effect to pausing (no new starts; in-flight runs finish). Option B: Zero-downtime “blue/green” with versioned queues (no scheduling pause) - Create a new queue for the new image (or even a new pool if you prefer strict isolation):

prefect work-queue create "process-v2" -p "<your-pool>"

- Start new workers (new image) pointing only to the new queue:

prefect worker start -p "<your-pool>" -q "process-v2" --with-healthcheck

- Update your deployments to target the new queue (via UI or

prefect deploy

) so new runs go to v2. - Pause the old queue to stop new work from going to old workers:

prefect work-queue pause "process-v1" -p "<your-pool>"

- Wait for old runs to finish, then shut down the old worker containers and remove the old queue. Health checks for safer rollouts (highly recommended) - Start workers with a health endpoint so Azure Container Apps can verify readiness:

prefect worker start -p "<your-pool>" --with-healthcheck

- Expose the web server in the container: - Set env vars so the health endpoint is reachable by ACA:

PREFECT_WORKER_WEBSERVER_HOST=0.0.0.0
    PREFECT_WORKER_WEBSERVER_PORT=8080

- Configure an ACA readiness probe to GET /health on that port. - Docs: Worker healthchecks A few practical tips - Prefer pausing/limiting on the queue(s) you use rather than the entire pool if you want finer control. - Keep the platform’s termination grace period long enough to let workers shut down cleanly after drains (ACA supports graceful termination of revisions). - If you’ve set a large worker prefetch window, consider lowering

PREFECT_WORKER_PREFETCH_SECONDS

during the drain to reduce pending “reserved” work. - Reminder: if a Process worker container is terminated while a run is in-flight, that run will be interrupted. Draining first avoids this. Relevant docs - Work pools - Workers - Worker healthchecks If you can share: - Your current pool/queue layout - Whether a brief scheduling pause is acceptable I can tailor an exact cutover script for ACA (including az containerapp commands) to make this fully hands-off.

Perfect — since you don’t want a scheduling pause and you already have two pools (pool_1 and pool_2), you can do true blue/green rollouts by flipping your deployments between the pools. This guarantees new runs only go to the new image while in-flight runs on the old image finish naturally. Recommended rollout pattern (no pause, no queues needed) - Current steady state: deployments target pool_1; workers for pool_1 are running on image v1. - Upgrade steps: 1) Bring up new workers on pool_2 with the new image (v2) - In Azure Container Apps, create/activate a new revision using the v2 image and command:

prefect worker start -p pool_2 --with-healthcheck

- Expose the health endpoint for ACA readiness checks: - Set env vars in the container:

PREFECT_WORKER_WEBSERVER_HOST=0.0.0.0
         PREFECT_WORKER_WEBSERVER_PORT=8080

- Configure ACA readiness probe to GET /health on port 8080. - Docs: Worker healthchecks 2) Update your deployments to target pool_2 - Since you deploy in Python, parameterize the pool name and redeploy:

import os
       from prefect.flows import Flow

       TARGET_POOL = os.getenv("TARGET_POOL", "pool_2")

       Flow.from_source(
           source=".",  # or your repo URL or storage
           entrypoint="path/to/flow.py:my_flow",
       ).deploy(
           name="my-deployment",
           work_pool_name=TARGET_POOL,
           # work_queue_name can be omitted since you're using the default
           # image=...  # optional, if you’re building images per-deployment
       )

API reference: Flow.deploy - From this point forward, all newly scheduled/triggered runs go to pool_2 (v2). No scheduling pause needed. 3) Let pool_1 drain naturally - Existing runs on pool_1 will continue and complete on the old workers. - You can watch the Work Pool page in the UI to see active runs for pool_1, or use:

prefect work-pool inspect "pool_1"

4) When pool_1 is idle, scale down/remove the pool_1 worker ACA revision - This completes the zero-downtime cutover. Next upgrade: flip back to pool_1 - Repeat the same steps, but bring up v3 workers on pool_1 and re-deploy flows with

work_pool_name="pool_1"

. This lets you alternate pools each release. Why this works - Without queues, workers in the same pool are indistinguishable to the scheduler — so to keep new runs off the old image without pausing, you need routing separation. Using two pools gives you that separation while allowing old runs to finish. Optional refinements - If you’d prefer to stay in a single pool, you can instead introduce two queues (e.g., default-v1, default-v2) and flip deployments between queues — but since you already have pool_1/pool_2, your current setup is simpler. - Keep

--with-healthcheck

on workers and use ACA readiness probes so traffic only shifts when the new workers are healthy. Useful references - Work pools: Work pools - Workers overview: Workers - Healthchecks: Worker healthchecks - Flow.deploy API: Flow.deploy If you share how your ACA app is set up (one container app per pool? revisions per image?), I can draft a concrete GitHub Actions job that: - Builds/pushes the new image - Activates a new ACA revision for pool_2 with healthchecks - Re-runs your Python deploy step targeting pool_2 - Optionally verifies pool_1 has drained before disabling the old revision

Got it — since your pools represent different resource classes (e.g., default vs high-memory), you don’t want to flip between pools. The cleanest zero-downtime pattern is to introduce versioned work queues inside each pool and route new runs to the new queue while old workers finish their in-flight runs. Pattern: per-pool blue/green with queues (no scheduling pause) For each pool (e.g., default and highmem): 1) Create a “v2” queue in the same pool - Example:

prefect work-queue create "default-v2" -p "pool_default"
  prefect work-queue create "highmem-v2" -p "pool_highmem"

2) Start new workers (new image) that only pull the v2 queue - In Azure Container Apps, create/activate a new revision for each pool’s worker using your new image and command:

prefect worker start -p pool_default -q default-v2 --with-healthcheck
  prefect worker start -p pool_highmem -q highmem-v2 --with-healthcheck

- Expose the health endpoint for ACA readiness:

PREFECT_WORKER_WEBSERVER_HOST=0.0.0.0
  PREFECT_WORKER_WEBSERVER_PORT=8080

Configure ACA readiness to GET /health on port 8080. Docs: Worker healthchecks 3) Update deployments to target the v2 queues - For “default” workloads:

Flow.from_source(
      source=".",  # your project/repo
      entrypoint="path/to/flow.py:my_flow",
  ).deploy(
      name="my-deployment",
      work_pool_name="pool_default",
      work_queue_name="default-v2",
  )

- For high-memory workloads:

Flow.from_source(...).deploy(
      name="my-highmem-deployment",
      work_pool_name="pool_highmem",
      work_queue_name="highmem-v2",
  )

API ref: Flow.deploy 4) Let old queues drain, then retire old workers - Old workers keep polling the old queue (“default” or “highmem”) and finish any in-flight runs. - When you see no remaining RUNNING runs on the old queue in the UI, scale down/disable the old ACA revisions. - No scheduling pause is needed; new runs have been routed to v2 the whole time. Why this works - The pool keeps its resource semantics. - The queue gives you routing control per release. New runs go to v2 queues and image v2 workers. Old runs complete on the old image. - You never terminate a worker that’s doing work — you only shut down the old revision after the old queue is empty. CI/CD tip - Parameterize the queue name per release, e.g.,

TARGET_QUEUE=default-v2

or

highmem-v2

, and pass it into your Python deployment step so your GitHub Action flips the queue for all relevant deployments. - If you want, you can alternate queue names (v1/v2) each release to keep the process consistent. References - Work pools - Workers - Worker healthchecks If you share the names of your pools and which deployments map to each, I can sketch a minimal GitHub Actions snippet that: - Builds/pushes the new image - Spins up ACA revisions pointing to the right queue per pool - Re-runs your Python deploy to set

work_queue_name

accordingly - Optionally verifies the old queues are drained before disabling the old revisions.

Short answer: you won’t get duplicate execution or true “race conditions,” but there isn’t a built‑in/supported way to bulk “move” existing SCHEDULED runs from one queue to another. Here’s what to know: - How queues work in Prefect 3.x - The deployment sets

work_pool_name

and optionally

work_queue_name

. New flow runs created for that deployment inherit that routing. - A run’s queue association is effectively fixed at creation time. Changing the deployment’s queue routes future runs, but does not retroactively move existing SCHEDULED runs. - Will moving jobs cause races? - If you briefly have both v1 and v2 queues “able” to see the same runs (e.g., overlapping filters), Prefect’s orchestration ensures only one worker can claim a run: the state transition to RUNNING is atomic. So you won’t get double execution. - The downside is operational: contention and noisy polling if both queues match the same runs — not correctness issues. - What you can safely do - Recommended: route new runs to the v2 queue (by updating the deployment) and let v1 drain. No pause needed; in-flight and already-scheduled v1 runs finish on v1. - If you must “move” existing SCHEDULED runs to v2: - There’s no bulk move command. The safe pattern is: 1) Update deployment(s) to

work_queue_name="...-v2"

for future runs. 2) For existing SCHEDULED runs on v1, cancel them and re-trigger new runs (now routed to v2). Do not terminate v1 workers until you’ve canceled/recreated any you care to migrate. - This avoids duplication, because you cancel the original runs before/when creating the replacements. - What not to do - Don’t shut down the v1 workers before v1’s SCHEDULED runs are handled; those runs will remain unclaimed if they’re still associated to the v1 queue. If you want, I can sketch a small script that: - Lists SCHEDULED runs for a given deployment (or pool/queue), - Cancels them, - Re-triggers new runs (which will go to the v2 queue after you update the deployment), so you can migrate a batch cleanly. References: - Deployments - Work pools and queues - Workers

Short answer: Yes — you can delete old queues after they’re drained. It won’t cause race conditions or duplicate runs, but make sure no deployments still reference those queues. Key points and caveats - Safe to delete once drained: - Deleting a queue does not affect runs that are already RUNNING. - The only risk is orphaning SCHEDULED runs that still target the old queue — they won’t be picked up if the queue is gone. - Update deployments first: - Redeploy or update each deployment to use the new queue (your git-hash queue) before deleting the old queue. - New runs will target the new queue; existing SCHEDULED runs on the old queue can either finish there or be cancelled and re-triggered. - Scope workers to the new queue: - Start workers with

-q <new-queue>

to avoid ambiguity and ensure they only poll the intended queue. Helpful commands - Verify the queue is empty before deleting:

prefect work-queue read-runs "<old-queue>" -p "<pool>"

or use the UI to confirm no SCHEDULED/RUNNING runs remain. - Delete the queue:

prefect work-queue delete "<old-queue>" -p "<pool>"

Operational tips for “queue-per-git-hash” - Garbage collect old queues: your approach will create many queues over time; periodically delete the old ones after drain. - Existing scheduled instances: if you need every run to execute on the new image, cancel SCHEDULED runs on the old queue and re-trigger them after deployments point to the new queue. - Health checks: keep

--with-healthcheck

on workers so ACA only routes to healthy revisions. References - Work pools and queues - Workers If you want, I can provide a tiny script or GH Action step that: - Lists SCHEDULED runs on an old queue, - Cancels them, - Re-triggers new runs (now routed to your git-hash queue), - Then deletes the old queue once empty.