Sean Talia

12/07/2020, 5:42 PM
After having combed through a good amount of documentation and a handful of threads in this channel, I'm having a hard time understanding what an appropriate use case of a
Docker Task
is; it seems like if you want to string together a flow that pulls an arbitrary set of docker images and runs them as containers in some sequence, the canonical way to do this is to make each instance of pulling + running a container its own
(that leverages docker
), and then to orchestrate a flow of flows. Given this, in what circumstance would you actually want to use a Docker Task? On the face of it, if you wanted to use one, it seems like you'd have to use some form of storage other than Docker for the task's parent flow to avoid a Docker-in-Docker situation; and I think not being able to use Docker Storage also implies that you wouldn't be able to use a Docker Agent for executing that flow + task? Does that sound right, or am I confusing the intended relationship between agent <-> flow <-> task?

Kyle Moon-Wright

12/07/2020, 6:13 PM
Hey @Sean Talia, I think this is a great summary of orchestrating with Docker Storage and there are certainly use cases for the Docker Tasks, check out these Docker Pipelines using the functional and imperative APIs - and the Docker Sidecar for Kubernetes deployment recipe. Additionally, using the orchestrator pattern you mentioned, you aren’t running a container within a container - you’re calling out to the backend API with the StartFlowRun task to run a flow which retrieves it’s own flow image. So storing your orchestrator flow using Docker Storage is perfectly valid. In general, I think it’s best to think of the Docker Tasks as a set of tools for the granular application of Docker functionality, to parse the process at a task level with Prefect’s configurations/functionality.