https://prefect.io
Join Slack
Hi. I was wondering if there is an example of running a flow with docker tasks on a fargate agent. I...
# ask-community
j
Hi. I was wondering if there is an example of running a flow with docker tasks on a fargate agent. I can get the docker tasks to run locally, and I can get non-docker tasks to run on fargate, but not both. (I can’t even get docker tasks to run on a local docker agent)
n
Hi @Joseph Solomon - could you provide some more context to your question? Is there a specific error you're running into?
j
yeah, i get errors like this
Copy code
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
which to me implies that there is no available docker daemon to connect to
n
That'd be my guess as well - which tracks, I think, because I don't think you should have access to the docker daemon from within Fargate. There's an issue on the aws roadmap here: https://github.com/aws/containers-roadmap/issues/95
j
I still wasn’t able to get the docker tasks to run on my docker agent locally either, any ideas how I would configure that to work?
n
Can you confirm the docker daemon is available and are you running any of this within a docker as well?
j
the docker daemon is available as the flow runs in docker, but I don’t know how to configure the flow to allow for docker tasks to run in the docker flow
n
@Joseph Solomon unless docker is installed inside the container or you've exposed the docker ports from the host machine to the container, the host machine's docker daemon won't be available to the individual tasks
j
yup, i understand that. I just don’t understand how to do that with my flow. if I set my flow to a Docker storage layer, how do I expose the docker ports when I run that flow?
n
Ah ok @Joseph Solomon I misunderstood - have you exposed the correct ports on the dockerfile you're using? (or are the correct ports exposed on the image you're using?)
If you're not using a dockerfile, you'll want to do so, even if just to extend an existing one (like 
prefecthq/prefect
) to expose the ports in question.
s
just flagging that I am dealing with precisely the same issue here (i.e. trying to use a docker agent + storage to run a flow whose individual tasks run docker containers), this might be something worth addressing in the documentation
@nicholas when you say "if you're not using a dockerfile, you'll want to do so", where exactly would the dockerfile be used? I have an agent that's been spun up with 
prefect agent docker start
, a flow that's been configured to use 
Docker()
storage as part of its run configuration, and a single task within that flow that just creates and starts a container of the 
hello-world
docker image; where in this context does a custom-written Dockerfile come into play?
(to be explicit, I'm not running 
prefect agent docker start
within a docker container, I'm running that just off my local machine that has prefect installed on it)
n
@Sean Talia - when you define Docker as the storage for your flow, you're implicitly building your flow based on the 
prefecthq/prefect
Docker image. As an alternative, you can choose to provide your own custom image (which can extend the base prefect one as well)
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by