https://prefect.io logo
j

Joseph Solomon

11/13/2020, 9:06 PM
Hi. I was wondering if there is an example of running a flow with docker tasks on a fargate agent. I can get the docker tasks to run locally, and I can get non-docker tasks to run on fargate, but not both. (I can’t even get docker tasks to run on a local docker agent)
n

nicholas

11/13/2020, 9:16 PM
Hi @Joseph Solomon - could you provide some more context to your question? Is there a specific error you're running into?
j

Joseph Solomon

11/13/2020, 9:17 PM
yeah, i get errors like this
Copy code
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
which to me implies that there is no available docker daemon to connect to
n

nicholas

11/13/2020, 10:28 PM
That'd be my guess as well - which tracks, I think, because I don't think you should have access to the docker daemon from within Fargate. There's an issue on the aws roadmap here: https://github.com/aws/containers-roadmap/issues/95
j

Joseph Solomon

11/16/2020, 4:08 PM
I still wasn’t able to get the docker tasks to run on my docker agent locally either, any ideas how I would configure that to work?
n

nicholas

11/16/2020, 4:55 PM
Can you confirm the docker daemon is available and are you running any of this within a docker as well?
j

Joseph Solomon

11/16/2020, 6:01 PM
the docker daemon is available as the flow runs in docker, but I don’t know how to configure the flow to allow for docker tasks to run in the docker flow
n

nicholas

11/16/2020, 10:05 PM
@Joseph Solomon unless docker is installed inside the container or you've exposed the docker ports from the host machine to the container, the host machine's docker daemon won't be available to the individual tasks
j

Joseph Solomon

11/16/2020, 10:19 PM
yup, i understand that. I just don’t understand how to do that with my flow. if I set my flow to a Docker storage layer, how do I expose the docker ports when I run that flow?
n

nicholas

11/17/2020, 12:59 AM
Ah ok @Joseph Solomon I misunderstood - have you exposed the correct ports on the dockerfile you're using? (or are the correct ports exposed on the image you're using?)
If you're not using a dockerfile, you'll want to do so, even if just to extend an existing one (like
prefecthq/prefect
) to expose the ports in question.
s

Sean Talia

12/02/2020, 8:59 PM
just flagging that I am dealing with precisely the same issue here (i.e. trying to use a docker agent + storage to run a flow whose individual tasks run docker containers), this might be something worth addressing in the documentation
@nicholas when you say "if you're not using a dockerfile, you'll want to do so", where exactly would the dockerfile be used? I have an agent that's been spun up with
prefect agent docker start
, a flow that's been configured to use
Docker()
storage as part of its run configuration, and a single task within that flow that just creates and starts a container of the
hello-world
docker image; where in this context does a custom-written Dockerfile come into play?
(to be explicit, I'm not running
prefect agent docker start
within a docker container, I'm running that just off my local machine that has prefect installed on it)
n

nicholas

12/02/2020, 10:30 PM
@Sean Talia - when you define Docker as the storage for your flow, you're implicitly building your flow based on the
prefecthq/prefect
Docker image. As an alternative, you can choose to provide your own custom image (which can extend the base prefect one as well)
2 Views