https://prefect.io logo
Channels
Powered by
#prefect-community
Title
# prefect-community
g

George Shishorin

06/22/2022, 12:20 PM
Hello, community! I’m trying to build a flow with docker tasks. So below is some information: • prefect server • prefect docker and local agents are started • task with pulling image from hub.docker.com (it’s pulling ok) • task that create container • task that run container But when I run flow locally there is an error (screen). Could you please help with it? P.S. code in thread
Copy code
from prefect import Flow
from prefect.run_configs import DockerRun
from prefect.tasks.docker import (
    PullImage,
    CreateContainer,
    StartContainer,
)

image = PullImage(
    repository="georgeshsh/main",
    tag="latest")

create_container = CreateContainer(
    image_name="georgeshsh/main:latest")

start_container = StartContainer()


with Flow("docker-task-example", run_config=DockerRun(labels=["docker"])) as flow:
    container_id = create_container(image)
    started = start_container(container_id=container_id)

flow.run()
k

Kevin Kho

06/22/2022, 1:12 PM
This is because you are attempting to run Docker inside Docker. The inner Docker doesn’t have access to the Docker daemon. Maybe you can try mounting the sock to the agent
Copy code
prefect agent docker start --volume //var/run/docker.sock:/var/run/docker.sock
g

George Shishorin

06/22/2022, 2:51 PM
Hello, Kevin! Thank you! As I understood these tasks are not running inside docker container, because I just debug flow and run it from python console.
This flow should pull image, create container and run it. So I suppose it’s possible to make it via local agent.
Error is
k

Kevin Kho

06/22/2022, 2:53 PM
If you use DockerRun and Docker Agent, there will be a new container for the Flow execution and the Flow will run inside, and then you try to do CreateContainer and StartContainer inside that
g

George Shishorin

06/22/2022, 2:53 PM
Ah, ok! Thank you very much.
k

Kevin Kho

06/22/2022, 2:54 PM
Try local agent yeah. That will work I think
g

George Shishorin

06/22/2022, 2:55 PM
Tried.
Error seems the same, but adds 
docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.41/containers/create>: Bad Request ("invalid reference format")
k

Kevin Kho

06/22/2022, 2:57 PM
Can you try giving a container_name to CreateContainer?
🙌 1
g

George Shishorin

06/22/2022, 3:09 PM
Now it works! Thank you!!
It should look like this:
Copy code
from prefect import Flow
from prefect.run_configs import UniversalRun
from prefect.tasks.docker import (CreateContainer, PullImage, StartContainer)

image = PullImage(
    repository="georgeshsh/main",
    tag="latest")

create_container = CreateContainer(
    image_name="georgeshsh/main:latest")

start_container = StartContainer()

with Flow("docker-task-example",
        run_config=UniversalRun(labels=["local"])
) as flow:
    container_id = create_container(container_name="test_container")
    started = start_container(container_id=container_id)

flow.run()
k

Kevin Kho

06/22/2022, 3:12 PM
Ah ok. It was just passing None as the container name 🤦‍♂️
g

George Shishorin

06/22/2022, 3:12 PM
Maybe add some defaults?
image name, f.e.
k

Kevin Kho

06/22/2022, 3:14 PM
You’re welcome to make a PR, or to assert it’s not None 🙂 . But this is the only time I’ve seen this so I’m not too worried
4 Views
Powered by