Thread
#prefect-community
    Joseph Solomon

    Joseph Solomon

    1 year ago
    Hi. I was wondering if there is an example of running a flow with docker tasks on a fargate agent. I can get the docker tasks to run locally, and I can get non-docker tasks to run on fargate, but not both. (I can’t even get docker tasks to run on a local docker agent)
    nicholas

    nicholas

    1 year ago
    Hi @Joseph Solomon - could you provide some more context to your question? Is there a specific error you're running into?
    Joseph Solomon

    Joseph Solomon

    1 year ago
    yeah, i get errors like this
    docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    which to me implies that there is no available docker daemon to connect to
    nicholas

    nicholas

    1 year ago
    That'd be my guess as well - which tracks, I think, because I don't think you should have access to the docker daemon from within Fargate. There's an issue on the aws roadmap here: https://github.com/aws/containers-roadmap/issues/95
    Joseph Solomon

    Joseph Solomon

    1 year ago
    I still wasn’t able to get the docker tasks to run on my docker agent locally either, any ideas how I would configure that to work?
    nicholas

    nicholas

    1 year ago
    Can you confirm the docker daemon is available and are you running any of this within a docker as well?
    Joseph Solomon

    Joseph Solomon

    1 year ago
    the docker daemon is available as the flow runs in docker, but I don’t know how to configure the flow to allow for docker tasks to run in the docker flow
    nicholas

    nicholas

    1 year ago
    @Joseph Solomon unless docker is installed inside the container or you've exposed the docker ports from the host machine to the container, the host machine's docker daemon won't be available to the individual tasks
    Joseph Solomon

    Joseph Solomon

    1 year ago
    yup, i understand that. I just don’t understand how to do that with my flow. if I set my flow to a Docker storage layer, how do I expose the docker ports when I run that flow?
    nicholas

    nicholas

    1 year ago
    Ah ok @Joseph Solomon I misunderstood - have you exposed the correct ports on the dockerfile you're using? (or are the correct ports exposed on the image you're using?)
    If you're not using a dockerfile, you'll want to do so, even if just to extend an existing one (like
    prefecthq/prefect
    ) to expose the ports in question.
    Sean Talia

    Sean Talia

    1 year ago
    just flagging that I am dealing with precisely the same issue here (i.e. trying to use a docker agent + storage to run a flow whose individual tasks run docker containers), this might be something worth addressing in the documentation
    @nicholas when you say "if you're not using a dockerfile, you'll want to do so", where exactly would the dockerfile be used? I have an agent that's been spun up with
    prefect agent docker start
    , a flow that's been configured to use
    Docker()
    storage as part of its run configuration, and a single task within that flow that just creates and starts a container of the
    hello-world
    docker image; where in this context does a custom-written Dockerfile come into play?
    (to be explicit, I'm not running
    prefect agent docker start
    within a docker container, I'm running that just off my local machine that has prefect installed on it)
    nicholas

    nicholas

    1 year ago
    @Sean Talia - when you define Docker as the storage for your flow, you're implicitly building your flow based on the
    prefecthq/prefect
    Docker image. As an alternative, you can choose to provide your own custom image (which can extend the base prefect one as well)