This message was deleted.
# ask-marvin
s
This message was deleted.
m
thought for 425.0 seconds
Great question. In Prefect 3.x there are two ways to set Docker job options like
auto_remove
without using the UI: 1) Per-deployment (simplest): pass
job_variables
when you deploy the flow 2) Pool-wide default: customize the work pool’s base job template (JSON) and set the default for
auto_remove
Here’s how to do both. Option 1 — Per-deployment (code only) This sets
auto_remove
for a specific deployment; no UI needed.
Copy code
from prefect import flow

@flow
def my_flow():
    ...

if __name__ == "__main__":
    my_flow.deploy(
        name="my-deployment",
        work_pool_name="my-docker-pool",
        job_variables={
            "auto_remove": True,
        },
    )
Option 2 — Pool-wide default by editing the base job template in code The base job template is a JSON object with two keys:
job_configuration
and
variables
. To make
auto_remove
default to true for all jobs submitted to the pool, set
variables.properties.auto_remove.default = true
. - If the pool already exists, fetch its template, update it, then save it back:
Copy code
import asyncio
from prefect.client.orchestration import get_client

async def main():
    async with get_client() as client:
        pool = await client.read_work_pool("my-docker-pool")
        tpl = pool.base_job_template

        # Ensure the variable exists, then change its default
        tpl["variables"]["properties"]["auto_remove"]["default"] = True

        await client.update_work_pool(
            name=pool.name,
            base_job_template=tpl,
        )

asyncio.run(main())
- If you are creating a new pool and want to do it entirely in code, you can start with the default Docker job configuration and set
auto_remove
as a default. For example:
Copy code
import asyncio
from prefect.client.orchestration import get_client

async def main():
    template = {
        "job_configuration": {
            "command": "{{ command }}",
            "env": "{{ env }}",
            "labels": "{{ labels }}",
            "name": "{{ name }}",
            "image": "{{ image }}",
            "registry_credentials": "{{ registry_credentials }}",
            "image_pull_policy": "{{ image_pull_policy }}",
            "networks": "{{ networks }}",
            "network_mode": "{{ network_mode }}",
            "auto_remove": "{{ auto_remove }}",
            "volumes": "{{ volumes }}",
            "stream_output": "{{ stream_output }}",
            "mem_limit": "{{ mem_limit }}",
            "memswap_limit": "{{ memswap_limit }}",
            "privileged": "{{ privileged }}",
            "container_create_kwargs": "{{ container_create_kwargs }}"
        },
        "variables": {
            "type": "object",
            "properties": {
                # define only what you want to change; the worker will use defaults for the rest
                "auto_remove": {
                    "type": "boolean",
                    "default": True,
                    "title": "Auto-remove container on exit"
                }
            }
        }
    }

    async with get_client() as client:
        await client.create_work_pool(
            name="my-docker-pool",
            type="docker",
            base_job_template=template,
        )

asyncio.run(main())
CLI alternative (still “code,” no UI) If you prefer to keep the full JSON template in your repo, you can: - Export the default template - Edit the JSON to set
variables.properties.auto_remove.default
to
true
- Create or update the pool using that file
Copy code
prefect work-pool get-default-base-job-template --type docker > docker-template.json
# edit docker-template.json so that:
#   "variables": { "properties": { "auto_remove": { "default": true, ... } } }
prefect work-pool create "my-docker-pool" --type docker --base-job-template ./docker-template.json
# or update an existing pool
prefect work-pool update "my-docker-pool" --base-job-template ./docker-template.json
Notes
- Pool-wide defaults set via the base job template apply to new flow runs picked up after the change; you don’t need to restart a running worker. - A deployment’s
job_variables
will override pool defaults when both are set. - If your current template doesn’t include
auto_remove
in
variables.properties
, just add that property as shown above with
default: true
. If you’d like, I can help you generate a ready-to-use
docker-template.json
with
auto_remove
defaulted to true.