Hi everyone, my first post here as I am new to Prefect. For my use-case, I am trying to construct a workflow running on a FargateCluster that reads a master file containing a list of large files (from S3) then map over each large file with several custom Docker images and Python tasks. However, we haven't been able to get even a simple Docker image task running (see below), the CreateContainer task results in a DockerException "No such file or directory". Any help would be greatly appreciated.

cluster = FargateCluster(
    image="<http://xxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/prefect-dask:latest|xxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/prefect-dask:latest>",
    scheduler_cpu=1024,
    scheduler_mem=4096,
    worker_cpu=256,
    worker_mem=512,
    cloudwatch_logs_group="prefect-dask-test",
    task_role_policies=['arn:aws:iam::aws:policy/AmazonS3FullAccess',
                        'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess']
)

# Prefect task definitions
create_container = CreateContainer(
    image_name="ubuntu",
    command="ls",
    volumes=['/var/run/docker.sock', '/var/run/docker.sock']
)
start = StartContainer()
logs = GetContainerLogs()
wait = WaitOnContainer()

# Prefect Flow definition
with Flow("Prefect Test Workflow") as flow:
    container_id = create_container()
    start = start(container_id=container_id)
    wait = wait(container_id=container_id)
    logs = logs(container_id=container_id)
    logs.set_upstream(wait)

state = flow.run(executor=DaskExecutor(address=cluster.scheduler_address))

Error encountered:
[2020-10-14 22:45:03] INFO - prefect.TaskRunner | Task 'CreateContainer': Starting task run...
[2020-10-14 22:45:03] ERROR - prefect.TaskRunner | Unexpected error: DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))")

Any help would be greatly appreciated for how to correctly run Docker images as tasks within Prefect on FargateCluster.

Hi @Ognjen Nikolic - welcome to Prefect! 👋 Sorry you're having some trouble here, let's see if we can figure out what's up. First thing,

DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))")

is a Docker-specific error that's usually a result of the Docker service/daemon not being available. Can you confirm the docker daemon is running on your local machine by running something like

docker ps

from your command line?

Thanks Nicholas for your prompt response. We were able to run the flow locally, we run into problems with Prefect running with DaskExecutor using the FargateCluster. We run the FargateCluster with our own base docker image, which has docker installed and we mount the docker socket in the create container command. Notice below when the base image is run with -v volumes flag, that docker is running while without that flag, docker daemon is not running. We tried passing in "volumes=['/var/run/docker.sock', '/var/run/docker.sock']" in the CreateContainer task but that did not seem to work, the task worker seems to not have the docker daemon running.

ubuntu@ip-172-31-11-141:~/prefect$ docker run -ti --entrypoint /bin/bash -v /var/run/docker.sock:/var/run/docker.sock prefect-dask:latest
root@39c81c103d3f:/# docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
39c81c103d3f        prefect-dask:latest   "/bin/bash"         3 seconds ago       Up 2 seconds                            serene_lichterman

ubuntu@ip-172-31-11-141:~/prefect$ docker run -ti --entrypoint /bin/bash prefect-dask:latest
root@b5f99f5ea304:/# docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Hi @Ognjen Nikolic - apologies for the slow reply. I have a suspicion that the

docker-py

create container method might not be starting the daemon as your CI command is doing; is it possible for you to ensure the Docker service is started in your container ahead of time (maybe an upstream task could handle this?) and we can test from there?

thanks Nicholas. I think what we are looking for is a AWS FargateCluster deployment similar to Docker Sidecar on Kubernetes described in this Prefect doc: https://docs.prefect.io/orchestration/recipes/k8s_docker_sidecar.html Do you know if deploying Prefect on Fargate is possible with Docker sidecar / Docker-in-Docker tasks?

I can’t get Fargate setup right now to try something out for you, but perhaps just trying to replicate the sidecar model from our example will work. Here’s a Fargate docker sidecar https://github.com/aws-samples/aws-xray-fargate/blob/master/src/service-a-taskdef.json