@Marvin when using a Docker Worker, can I automatically prune the container after it has run?
My local file system is bloated by used containers.
m
Marvin
02/27/2024, 10:24 PM
I couldn't find specific documentation on automatically pruning Docker containers after they have run when using a Docker Worker in Prefect. Pruning containers is generally a manual process or something you script yourself, but you might want to look into the Docker settings or your worker configuration to see if there's an existing option for cleanup after a job is done.
In Docker, you can manually prune containers using the
docker container prune
command. Alternatively, you can use the
--rm
flag when running a container to automatically remove it when it finishes. If Prefect's Docker Agent or Docker Runner doesn't support auto-pruning now, you may have to run a cleanup command yourself after your workflows complete or on a schedule.
You might also want to explore whether Prefect exposes any Docker-related configuration options in code or through the CLI and check if there's something in there about cleanup after runs.
It might be worth searching the GitHub issues for any relevant discussions or feature requests related to this.
Unfortunately, it seems dealing with a growing number of old containers is just another thing in my vast repertoire of pain and sorrow.
n
Nate
02/27/2024, 10:24 PM
Nate
02/27/2024, 10:24 PM
on the docker work pool
b
Baker
02/27/2024, 10:25 PM
Thanks Nate for always watching Marvin and jumping in 👍
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.