Short answer: yes, we’ve seen Azure Container Instances occasionally stick around after successful runs when used via the ACI worker in a hybrid pool. The ACI worker deletes the container group at the end of a run, but if deletion takes too long or the worker exits before the cleanup finishes, you can end up with orphaned container groups. Key details and what to check - In Prefect 3.x, the ACI worker’s default behavior is to delete the container group after completion. That’s controlled by a boolean setting

keep_container_group

(default false). If it’s true, groups are stopped (not deleted). - The worker waits up to ~30 seconds for deletion. If Azure takes longer, the worker raises a timeout and you may see orphaned groups as a result. - If the worker crashes or is terminated before the cleanup “finally” block runs, deletion won’t occur. - There are no tags added to ACI resources by default, so ad-hoc cleanup in Azure can be harder unless you customize the job template. Docs - ACI worker guide: ACI worker (Prefect + Azure) - Work pool CLI reference: Work pool CLI (Prefect v3) Immediate triage - Confirm your pool’s job template has

keep_container_group: false

. - View the configured template:

prefect work-pool inspect "<your-pool>"

- See the default ACI template for comparison:

prefect work-pool get-default-base-job-template --type azure-container-instance

- Check worker logs near the end of a flow run for messages like “Timed out while waiting for deletion of container group …”. If you see those, you are likely hitting the Azure-side delay that leaves the group around. - Confirm the identity used by the worker has permission to delete ACI container groups in the target resource group (e.g., Container Instance Contributor or equivalent custom role). - Ensure the worker process is not being stopped/killed too aggressively by your hosting environment right after runs finish. Recommended workarounds 1) Add tags to ACI resources via your base job template, then clean up by tag - You can customize the ACI ARM template in your work pool’s base job template to add tags for easy filtering (for example, managed-by=prefect, cleanup-eligible=true). Retrieve the current template, add a tags block, then update the pool:

prefect work-pool get-default-base-job-template --type azure-container-instance > aci-template.json
  # Edit aci-template.json — in arm_template.resources[0], add:
  # "tags": { "managed-by": "prefect", "cleanup-eligible": "true" }
  prefect work-pool update "<your-pool>" --base-job-template aci-template.json

After that, everything created by this pool will carry those tags. 2) Schedule a periodic cleanup job - Run a small scheduled Prefect flow or a cron/automation job that: - Lists container groups with your cleanup tag - Checks their finish time - Deletes anything older than a TTL you choose (e.g., 24 hours) - A simple Azure CLI-based approach: ``` #!/usr/bin/env bash set -euo pipefail RG="your-resource-group" TTL_HOURS=24 TAG_KEY="cleanup-eligible" TAG_VALUE="true" now=$(date -u +%s) # List candidates by tag; for each, fetch finish time and delete if older than TTL az container list -g "$RG" \ --query "[?tags.$TAG_KEY=='$TAG_VALUE'].name" -o tsv | while read -r NAME; do # Fetch the container's finish time (if present) FINISH_TIME=$(az container show -g "$RG" -n "$NAME" \ --query "containers[0].instanceView.currentState.finishTime" -o tsv || echo "") # If finish time is missing, skip or optionally fall back to startTime if [ -z "${FINISH_TIME}" ] || [ "${FINISH_TIME}" == "null" ]; then continue fi finish_ts=$(date -u -d "$FINISH_TIME" +%s || echo 0) age_hours=$(( (now - finish_ts) / 3600 )) if [ "$age_hours" -ge "$TTL_HOURS" ]; then

echo "Deleting $NAME (finished ${age_hours}h ago)" az container delete -g "$RG" -n "$NAME" --yes fi done ``` You can wrap this in a small Prefect flow and put it on a schedule, or run it in your preferred scheduler. 3) Reduce the chance of orphans - Keep

keep_container_group=false

for production runs. - Avoid terminating the worker immediately after runs finish; give it time to perform cleanup. - If you run heavy parallel bursts, consider modest rate limiting to avoid Azure API/propagation delays during create/delete storms. If you’d like, I can help you: - Inspect your pool’s base job template and add tags + confirm

keep_container_group

is false - Draft a small Prefect flow you can schedule to run the cleanup routine - Look at a snippet of your worker logs around one of the runs where cleanup failed A couple of quick questions to tailor the solution: - What’s the name of the work pool and is it using the ACI worker type within your hybrid pool (type

azure-container-instance

)? - Do you see timeout messages in the worker logs when the deletion should occur? - Which resource group(s) and region(s) are you running in, and are you using a managed identity for the ACI operations?

Short answer: not today. In Prefect 3.x’s ACI worker the deletion wait is hard-coded to about 30 seconds. There isn’t a user-facing setting to increase it, so occasionally-slow Azure deletions can leave behind orphaned container groups even when the flow run succeeds. Workarounds you can use right now - Periodic cleanup job in your Prefect-only RG - Since your resource group is dedicated to Prefect, a simple scheduled cleanup is the safest/least invasive option. For example, delete any ACI container groups that are finished/terminated and older than a small buffer (e.g., 15–30 minutes):

# delete terminated containers older than 30 minutes
  RG="your-prefect-rg"
  CUTOFF="$(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%SZ)"

  az container list -g "$RG" \
    --query "[?containers[0].instanceView.currentState.state=='Terminated' && containers[0].instanceView.currentState.finishTime < '$CUTOFF'].[name]" \
    -o tsv | while read -r NAME; do
      echo "Deleting $NAME"
      az container delete -g "$RG" -n "$NAME" --yes
  done

You can run this via a small Prefect flow on a schedule or with your existing scheduler. - Vendor/override if you really need a longer wait - If you control the worker image, you could fork the ACI worker (prefect-azure) and bump the deletion timeout constant, then run your hybrid worker from that image. It’s a small change but does require maintaining a custom build. - Optional: reduce the likelihood of orphans - Avoid terminating the worker process right after runs finish so it can complete cleanup. - If you run large bursts, modest rate limiting can help avoid Azure-side delays during create/delete storms. If you’d like, I can: - Share a minimal Prefect flow you can deploy to run the cleanup on a schedule - Help you patch a custom worker image to increase the timeout Docs for reference - ACI worker guide: Prefect + Azure Container Instances Would you prefer the scheduled cleanup flow example, or are you interested in the custom worker route?