https://prefect.io
Join Slack
Hello, community! I’m trying to build a flow with docker tasks. So below is some information: - pre...
# ask-community
g
Hello, community! I’m trying to build a flow with docker tasks. So below is some information: • prefect server • prefect docker and local agents are started • task with pulling image from hub.docker.com (it’s pulling ok) • task that create container • task that run container But when I run flow locally there is an error (screen). Could you please help with it? P.S. code in thread
Copy code
from prefect import Flow
from prefect.run_configs import DockerRun
from prefect.tasks.docker import (
    PullImage,
    CreateContainer,
    StartContainer,
)

image = PullImage(
    repository="georgeshsh/main",
    tag="latest")

create_container = CreateContainer(
    image_name="georgeshsh/main:latest")

start_container = StartContainer()


with Flow("docker-task-example", run_config=DockerRun(labels=["docker"])) as flow:
    container_id = create_container(image)
    started = start_container(container_id=container_id)

flow.run()
k
This is because you are attempting to run Docker inside Docker. The inner Docker doesn’t have access to the Docker daemon. Maybe you can try mounting the sock to the agent
Copy code
prefect agent docker start --volume //var/run/docker.sock:/var/run/docker.sock
g
Hello, Kevin! Thank you! As I understood these tasks are not running inside docker container, because I just debug flow and run it from python console.
This flow should pull image, create container and run it. So I suppose it’s possible to make it via local agent.
Error is
k
If you use DockerRun and Docker Agent, there will be a new container for the Flow execution and the Flow will run inside, and then you try to do CreateContainer and StartContainer inside that
g
Ah, ok! Thank you very much.
k
Try local agent yeah. That will work I think
g
Tried.
Error seems the same, but adds 
docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.41/containers/create>: Bad Request ("invalid reference format")
k
Can you try giving a container_name to CreateContainer?
🙌 1
g
Now it works! Thank you!!
It should look like this:
Copy code
from prefect import Flow
from prefect.run_configs import UniversalRun
from prefect.tasks.docker import (CreateContainer, PullImage, StartContainer)

image = PullImage(
    repository="georgeshsh/main",
    tag="latest")

create_container = CreateContainer(
    image_name="georgeshsh/main:latest")

start_container = StartContainer()

with Flow("docker-task-example",
        run_config=UniversalRun(labels=["local"])
) as flow:
    container_id = create_container(container_name="test_container")
    started = start_container(container_id=container_id)

flow.run()
k
Ah ok. It was just passing None as the container name 🤦‍♂️
g
Maybe add some defaults?
image name, f.e.
k
You’re welcome to make a PR, or to assert it’s not None 🙂 . But this is the only time I’ve seen this so I’m not too worried
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by