Slackbot
09/09/2025, 8:30 AMMarvin
09/09/2025, 8:30 AMMarvin
09/09/2025, 8:37 AMauto_remove
without using the UI:
1) Per-deployment (simplest): pass job_variables
when you deploy the flow
2) Pool-wide default: customize the work pool’s base job template (JSON) and set the default for auto_remove
Here’s how to do both.
Option 1 — Per-deployment (code only)
This sets auto_remove
for a specific deployment; no UI needed.
from prefect import flow
@flow
def my_flow():
...
if __name__ == "__main__":
my_flow.deploy(
name="my-deployment",
work_pool_name="my-docker-pool",
job_variables={
"auto_remove": True,
},
)
Option 2 — Pool-wide default by editing the base job template in code
The base job template is a JSON object with two keys: job_configuration
and variables
. To make auto_remove
default to true for all jobs submitted to the pool, set variables.properties.auto_remove.default = true
.
- If the pool already exists, fetch its template, update it, then save it back:
import asyncio
from prefect.client.orchestration import get_client
async def main():
async with get_client() as client:
pool = await client.read_work_pool("my-docker-pool")
tpl = pool.base_job_template
# Ensure the variable exists, then change its default
tpl["variables"]["properties"]["auto_remove"]["default"] = True
await client.update_work_pool(
name=pool.name,
base_job_template=tpl,
)
asyncio.run(main())
- If you are creating a new pool and want to do it entirely in code, you can start with the default Docker job configuration and set auto_remove
as a default. For example:
import asyncio
from prefect.client.orchestration import get_client
async def main():
template = {
"job_configuration": {
"command": "{{ command }}",
"env": "{{ env }}",
"labels": "{{ labels }}",
"name": "{{ name }}",
"image": "{{ image }}",
"registry_credentials": "{{ registry_credentials }}",
"image_pull_policy": "{{ image_pull_policy }}",
"networks": "{{ networks }}",
"network_mode": "{{ network_mode }}",
"auto_remove": "{{ auto_remove }}",
"volumes": "{{ volumes }}",
"stream_output": "{{ stream_output }}",
"mem_limit": "{{ mem_limit }}",
"memswap_limit": "{{ memswap_limit }}",
"privileged": "{{ privileged }}",
"container_create_kwargs": "{{ container_create_kwargs }}"
},
"variables": {
"type": "object",
"properties": {
# define only what you want to change; the worker will use defaults for the rest
"auto_remove": {
"type": "boolean",
"default": True,
"title": "Auto-remove container on exit"
}
}
}
}
async with get_client() as client:
await client.create_work_pool(
name="my-docker-pool",
type="docker",
base_job_template=template,
)
asyncio.run(main())
CLI alternative (still “code,” no UI)
If you prefer to keep the full JSON template in your repo, you can:
- Export the default template
- Edit the JSON to set variables.properties.auto_remove.default
to true
- Create or update the pool using that file
prefect work-pool get-default-base-job-template --type docker > docker-template.json
# edit docker-template.json so that:
# "variables": { "properties": { "auto_remove": { "default": true, ... } } }
prefect work-pool create "my-docker-pool" --type docker --base-job-template ./docker-template.json
# or update an existing pool
prefect work-pool update "my-docker-pool" --base-job-template ./docker-template.json
NotesMarvin
09/09/2025, 8:37 AMjob_variables
will override pool defaults when both are set.
- If your current template doesn’t include auto_remove
in variables.properties
, just add that property as shown above with default: true
.
If you’d like, I can help you generate a ready-to-use docker-template.json
with auto_remove
defaulted to true.