Ognjen Nikolic

    Ognjen Nikolic

    1 year ago
    Hi everyone, my first post here as I am new to Prefect. For my use-case, I am trying to construct a workflow running on a FargateCluster that reads a master file containing a list of large files (from S3) then map over each large file with several custom Docker images and Python tasks. However, we haven't been able to get even a simple Docker image task running (see below), the CreateContainer task results in a DockerException "No such file or directory". Any help would be greatly appreciated.
    cluster = FargateCluster(
        image="<http://xxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/prefect-dask:latest|xxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/prefect-dask:latest>",
        scheduler_cpu=1024,
        scheduler_mem=4096,
        worker_cpu=256,
        worker_mem=512,
        cloudwatch_logs_group="prefect-dask-test",
        task_role_policies=['arn:aws:iam::aws:policy/AmazonS3FullAccess',
                            'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess']
    )
    
    # Prefect task definitions
    create_container = CreateContainer(
        image_name="ubuntu",
        command="ls",
        volumes=['/var/run/docker.sock', '/var/run/docker.sock']
    )
    start = StartContainer()
    logs = GetContainerLogs()
    wait = WaitOnContainer()
    
    # Prefect Flow definition
    with Flow("Prefect Test Workflow") as flow:
        container_id = create_container()
        start = start(container_id=container_id)
        wait = wait(container_id=container_id)
        logs = logs(container_id=container_id)
        logs.set_upstream(wait)
    
    state = flow.run(executor=DaskExecutor(address=cluster.scheduler_address))
    
    Error encountered:
    [2020-10-14 22:45:03] INFO - prefect.TaskRunner | Task 'CreateContainer': Starting task run...
    [2020-10-14 22:45:03] ERROR - prefect.TaskRunner | Unexpected error: DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))")
    Any help would be greatly appreciated for how to correctly run Docker images as tasks within Prefect on FargateCluster.
    nicholas

    nicholas

    1 year ago
    Hi @Ognjen Nikolic - welcome to Prefect! 👋 Sorry you're having some trouble here, let's see if we can figure out what's up. First thing,
    DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))")
    is a Docker-specific error that's usually a result of the Docker service/daemon not being available. Can you confirm the docker daemon is running on your local machine by running something like
    docker ps
    from your command line?
    Ognjen Nikolic

    Ognjen Nikolic

    1 year ago
    Thanks Nicholas for your prompt response. We were able to run the flow locally, we run into problems with Prefect running with DaskExecutor using the FargateCluster. We run the FargateCluster with our own base docker image, which has docker installed and we mount the docker socket in the create container command. Notice below when the base image is run with -v volumes flag, that docker is running while without that flag, docker daemon is not running. We tried passing in "volumes=['/var/run/docker.sock', '/var/run/docker.sock']" in the CreateContainer task but that did not seem to work, the task worker seems to not have the docker daemon running.
    ubuntu@ip-172-31-11-141:~/prefect$ docker run -ti --entrypoint /bin/bash -v /var/run/docker.sock:/var/run/docker.sock prefect-dask:latest
    root@39c81c103d3f:/# docker ps
    CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
    39c81c103d3f        prefect-dask:latest   "/bin/bash"         3 seconds ago       Up 2 seconds                            serene_lichterman
    
    ubuntu@ip-172-31-11-141:~/prefect$ docker run -ti --entrypoint /bin/bash prefect-dask:latest
    root@b5f99f5ea304:/# docker ps
    Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
    nicholas

    nicholas

    1 year ago
    Hi @Ognjen Nikolic - apologies for the slow reply. I have a suspicion that the
    docker-py
    create container method might not be starting the daemon as your CI command is doing; is it possible for you to ensure the Docker service is started in your container ahead of time (maybe an upstream task could handle this?) and we can test from there?
    Ognjen Nikolic

    Ognjen Nikolic

    1 year ago
    thanks Nicholas. I think what we are looking for is a AWS FargateCluster deployment similar to Docker Sidecar on Kubernetes described in this Prefect doc: https://docs.prefect.io/orchestration/recipes/k8s_docker_sidecar.html Do you know if deploying Prefect on Fargate is possible with Docker sidecar / Docker-in-Docker tasks?
    Michael Adkins

    Michael Adkins

    1 year ago
    I can’t get Fargate setup right now to try something out for you, but perhaps just trying to replicate the sidecar model from our example will work. Here’s a Fargate docker sidecar https://github.com/aws-samples/aws-xray-fargate/blob/master/src/service-a-taskdef.json