https://prefect.io logo
o

Ognjen Nikolic

10/14/2020, 11:01 PM
Hi everyone, my first post here as I am new to Prefect. For my use-case, I am trying to construct a workflow running on a FargateCluster that reads a master file containing a list of large files (from S3) then map over each large file with several custom Docker images and Python tasks. However, we haven't been able to get even a simple Docker image task running (see below), the CreateContainer task results in a DockerException "No such file or directory". Any help would be greatly appreciated.
Copy code
cluster = FargateCluster(
    image="<http://xxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/prefect-dask:latest|xxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/prefect-dask:latest>",
    scheduler_cpu=1024,
    scheduler_mem=4096,
    worker_cpu=256,
    worker_mem=512,
    cloudwatch_logs_group="prefect-dask-test",
    task_role_policies=['arn:aws:iam::aws:policy/AmazonS3FullAccess',
                        'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess']
)

# Prefect task definitions
create_container = CreateContainer(
    image_name="ubuntu",
    command="ls",
    volumes=['/var/run/docker.sock', '/var/run/docker.sock']
)
start = StartContainer()
logs = GetContainerLogs()
wait = WaitOnContainer()

# Prefect Flow definition
with Flow("Prefect Test Workflow") as flow:
    container_id = create_container()
    start = start(container_id=container_id)
    wait = wait(container_id=container_id)
    logs = logs(container_id=container_id)
    logs.set_upstream(wait)

state = flow.run(executor=DaskExecutor(address=cluster.scheduler_address))

Error encountered:
[2020-10-14 22:45:03] INFO - prefect.TaskRunner | Task 'CreateContainer': Starting task run...
[2020-10-14 22:45:03] ERROR - prefect.TaskRunner | Unexpected error: DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))")
Any help would be greatly appreciated for how to correctly run Docker images as tasks within Prefect on FargateCluster.
n

nicholas

10/14/2020, 11:14 PM
Hi @Ognjen Nikolic - welcome to Prefect! 👋 Sorry you're having some trouble here, let's see if we can figure out what's up. First thing,
DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))")
is a Docker-specific error that's usually a result of the Docker service/daemon not being available. Can you confirm the docker daemon is running on your local machine by running something like
docker ps
from your command line?
o

Ognjen Nikolic

10/14/2020, 11:29 PM
Thanks Nicholas for your prompt response. We were able to run the flow locally, we run into problems with Prefect running with DaskExecutor using the FargateCluster. We run the FargateCluster with our own base docker image, which has docker installed and we mount the docker socket in the create container command. Notice below when the base image is run with -v volumes flag, that docker is running while without that flag, docker daemon is not running. We tried passing in "volumes=['/var/run/docker.sock', '/var/run/docker.sock']" in the CreateContainer task but that did not seem to work, the task worker seems to not have the docker daemon running.
Copy code
ubuntu@ip-172-31-11-141:~/prefect$ docker run -ti --entrypoint /bin/bash -v /var/run/docker.sock:/var/run/docker.sock prefect-dask:latest
root@39c81c103d3f:/# docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
39c81c103d3f        prefect-dask:latest   "/bin/bash"         3 seconds ago       Up 2 seconds                            serene_lichterman

ubuntu@ip-172-31-11-141:~/prefect$ docker run -ti --entrypoint /bin/bash prefect-dask:latest
root@b5f99f5ea304:/# docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
n

nicholas

10/15/2020, 2:50 PM
Hi @Ognjen Nikolic - apologies for the slow reply. I have a suspicion that the
docker-py
create container method might not be starting the daemon as your CI command is doing; is it possible for you to ensure the Docker service is started in your container ahead of time (maybe an upstream task could handle this?) and we can test from there?
o

Ognjen Nikolic

10/15/2020, 11:47 PM
thanks Nicholas. I think what we are looking for is a AWS FargateCluster deployment similar to Docker Sidecar on Kubernetes described in this Prefect doc: https://docs.prefect.io/orchestration/recipes/k8s_docker_sidecar.html Do you know if deploying Prefect on Fargate is possible with Docker sidecar / Docker-in-Docker tasks?
z

Zanie

10/16/2020, 1:53 PM
I can’t get Fargate setup right now to try something out for you, but perhaps just trying to replicate the sidecar model from our example will work. Here’s a Fargate docker sidecar https://github.com/aws-samples/aws-xray-fargate/blob/master/src/service-a-taskdef.json
upvote 1