Hi I was wondering if there is an example of running a flow

Question

Hi I was wondering if there is an example of running a flow with docker tasks on a fargate agent I can get the docker tasks to run locally and I can get non docker tasks to run on fargate but not both I can t even get docker tasks to run on a local docker agent

nicholas · Answer

Hi < Joseph Solomon> could you provide some more context to your question Is there a specific error you re running into

Joseph Solomon · Answer

yeah i get errors like this ``` docker errors DockerException Error while fetching server API version Connection aborted FileNotFoundError 2 No such file or directory ``` which to me implies that there is no available docker daemon to connect to

nicholas · Answer

That d be my guess as well which tracks I think because I don t think you should have access to the docker daemon from within Fargate There s an issue on the aws roadmap here <https github com aws containers roadmap issues 95>

Joseph Solomon · Answer

I still wasn t able to get the docker tasks to run on my docker agent locally either any ideas how I would configure that to work

nicholas · Answer

Can you confirm the docker daemon is available and are you running any of this within a docker as well

Joseph Solomon · Answer

the docker daemon is available as the flow runs in docker but I don t know how to configure the flow to allow for docker tasks to run in the docker flow

nicholas · Answer

< Joseph Solomon> unless docker is installed inside the container or you ve exposed the docker ports from the host machine to the container the host machine s docker daemon won t be available to the individual tasks

Joseph Solomon · Answer

yup i understand that I just don t understand how to do that with my flow if I set my flow to a Docker storage layer how do I expose the docker ports when I run that flow

nicholas · Answer

Ah ok < Joseph Solomon> I misunderstood have you exposed the correct ports on the dockerfile you re using or are the correct ports exposed on the image you re using

nicholas · Answer

If you re not using a dockerfile you ll want to do so even if just to extend an existing one like `prefecthq prefect` to expose the ports in question

Sean Talia · Answer

just flagging that I am dealing with precisely the same issue here i e trying to use a docker agent + storage to run a flow whose individual tasks run docker containers this might be something worth addressing in the documentation

Sean Talia · Answer

< nicholas> when you say if you re not using a dockerfile you ll want to do so where exactly would the dockerfile be used I have an agent that s been spun up with `prefect agent docker start` a flow that s been configured to use `Docker ` storage as part of its run configuration and a single task within that flow that just creates and starts a container of the `hello world` docker image where in this context does a custom written Dockerfile come into play

Sean Talia · Answer

to be explicit I m not running `prefect agent docker start` within a docker container I m running that just off my local machine that has prefect installed on it

nicholas · Answer

< Sean Talia> when you define Docker as the storage for your flow you re implicitly building your flow based on the `prefecthq prefect` Docker image As an alternative you can choose to provide your own custom image which can extend the base prefect one as well