I am reading in the prefect logs here that you can pull an image, start a container with a command, wait for the container to finish running, then stop the container. This is 4 tasks, but it would be really nice if I could just add a docker image name as an argument to the task decorator.

Hi Zach, There's lots of ways to do this, it depends on what you want: • Do you want the task to run a CLI command inside a docker container? Then using a task to start a docker container is the recommended way. You could write your own task using the docker api to do the start and wait, or compose together a few tasks from the task library to do so (since prefect flows can be programmatically generated, you could write a helper to generate this pattern for you). • Do you have a small number of different containers, and just want to run a python function inside a specific container? Then you might make use of a

DaskExecutor

with workers running in individual containers, and use resource labels to lock tasks to run in a specific worker. This is a lot more complicated setup (a lot more complicated), but could also be useful in some scenarios. If possible, I recommend the first option. We could add a new task to the task library to create, run, and wait on a single docker container that would make this pattern easier, but it would still be for executing a CLI command inside a container not running a python function.

Having a task that packages together the create, run and, wait for a docker container would be great. It would also be helpful to have a way to copy a file into the docker container

Sure, I'll open an issue.

It would also be helpful to have a way to copy a file into the docker container

That seems like a bit of scope creep, I think for that I'd want the user to write their own custom thing - prefect works fine with user-written tasks, only common patterns make sense for inclusion in the task library.

Some pipelines, including many of the ones I work with, operate on files going through the tasks. So for a docker container task like this, I need a way to get the input file to the container.

I'd do this through volumes, which we would expose. So you could create a docker volume, then use this command to mount and write/read files in the volume.

@Jim Crist-Harif How would I go about running docker container task in a KubernetesJobEnvironment? Because you would need to be running a docker container from within a docker container

See https://docs.prefect.io/orchestration/recipes/k8s_docker_sidecar.html for a working example.