George Shishorin

    George Shishorin

    3 months ago
    Hello, community! I’m trying to build a flow with docker tasks. So below is some information: • prefect server • prefect docker and local agents are started • task with pulling image from hub.docker.com (it’s pulling ok) • task that create container • task that run container But when I run flow locally there is an error (screen). Could you please help with it? P.S. code in thread
    from prefect import Flow
    from prefect.run_configs import DockerRun
    from prefect.tasks.docker import (
        PullImage,
        CreateContainer,
        StartContainer,
    )
    
    image = PullImage(
        repository="georgeshsh/main",
        tag="latest")
    
    create_container = CreateContainer(
        image_name="georgeshsh/main:latest")
    
    start_container = StartContainer()
    
    
    with Flow("docker-task-example", run_config=DockerRun(labels=["docker"])) as flow:
        container_id = create_container(image)
        started = start_container(container_id=container_id)
    
    flow.run()
    Kevin Kho

    Kevin Kho

    3 months ago
    This is because you are attempting to run Docker inside Docker. The inner Docker doesn’t have access to the Docker daemon. Maybe you can try mounting the sock to the agent
    prefect agent docker start --volume //var/run/docker.sock:/var/run/docker.sock
    George Shishorin

    George Shishorin

    3 months ago
    Hello, Kevin! Thank you! As I understood these tasks are not running inside docker container, because I just debug flow and run it from python console.
    This flow should pull image, create container and run it. So I suppose it’s possible to make it via local agent.
    Error is
    Kevin Kho

    Kevin Kho

    3 months ago
    If you use DockerRun and Docker Agent, there will be a new container for the Flow execution and the Flow will run inside, and then you try to do CreateContainer and StartContainer inside that
    George Shishorin

    George Shishorin

    3 months ago
    Ah, ok! Thank you very much.
    Kevin Kho

    Kevin Kho

    3 months ago
    Try local agent yeah. That will work I think
    George Shishorin

    George Shishorin

    3 months ago
    Tried.
    Error seems the same, but adds
    docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.41/containers/create>: Bad Request ("invalid reference format")
    Kevin Kho

    Kevin Kho

    3 months ago
    Can you try giving a container_name to CreateContainer?
    George Shishorin

    George Shishorin

    3 months ago
    Now it works! Thank you!!
    It should look like this:
    from prefect import Flow
    from prefect.run_configs import UniversalRun
    from prefect.tasks.docker import (CreateContainer, PullImage, StartContainer)
    
    image = PullImage(
        repository="georgeshsh/main",
        tag="latest")
    
    create_container = CreateContainer(
        image_name="georgeshsh/main:latest")
    
    start_container = StartContainer()
    
    with Flow("docker-task-example",
            run_config=UniversalRun(labels=["local"])
    ) as flow:
        container_id = create_container(container_name="test_container")
        started = start_container(container_id=container_id)
    
    flow.run()
    Kevin Kho

    Kevin Kho

    3 months ago
    Ah ok. It was just passing None as the container name 🤦‍♂️
    George Shishorin

    George Shishorin

    3 months ago
    Maybe add some defaults?
    image name, f.e.
    Kevin Kho

    Kevin Kho

    3 months ago
    You’re welcome to make a PR, or to assert it’s not None 🙂 . But this is the only time I’ve seen this so I’m not too worried