Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
g
George Shishorin
06/22/2022, 12:20 PMHello, community!
I’m trying to build a flow with docker tasks. So below is some information:
• prefect server
• prefect docker and local agents are started
• task with pulling image from hub.docker.com (it’s pulling ok)
• task that create container
• task that run container
But when I run flow locally there is an error (screen).
Could you please help with it?
P.S. code in thread
from prefect import Flow
from prefect.run_configs import DockerRun
from prefect.tasks.docker import (
PullImage,
CreateContainer,
StartContainer,
)
image = PullImage(
repository="georgeshsh/main",
tag="latest")
create_container = CreateContainer(
image_name="georgeshsh/main:latest")
start_container = StartContainer()
with Flow("docker-task-example", run_config=DockerRun(labels=["docker"])) as flow:
container_id = create_container(image)
started = start_container(container_id=container_id)
flow.run()k
Kevin Kho
06/22/2022, 1:12 PMThis is because you are attempting to run Docker inside Docker. The inner Docker doesn’t have access to the Docker daemon. Maybe you can try mounting the sock to the agent
prefect agent docker start --volume //var/run/docker.sock:/var/run/docker.sockg
George Shishorin
06/22/2022, 2:51 PMHello, Kevin!
Thank you!
As I understood these tasks are not running inside docker container, because I just debug flow and run it from python console.
This flow should pull image, create container and run it. So I suppose it’s possible to make it via local agent.
Error is
k
Kevin Kho
06/22/2022, 2:53 PMIf you use DockerRun and Docker Agent, there will be a new container for the Flow execution and the Flow will run inside, and then you try to do CreateContainer and StartContainer inside that
g
George Shishorin
06/22/2022, 2:53 PMAh, ok! Thank you very much.
k
Kevin Kho
06/22/2022, 2:54 PMTry local agent yeah. That will work I think
g
George Shishorin
06/22/2022, 2:55 PMTried.
Error seems the same, but adds
docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.41/containers/create>: Bad Request ("invalid reference format")k
Kevin Kho
06/22/2022, 2:57 PMCan you try giving a container_name to CreateContainer?
🙌 1
g
George Shishorin
06/22/2022, 3:09 PMNow it works!
Thank you!!
It should look like this:
from prefect import Flow
from prefect.run_configs import UniversalRun
from prefect.tasks.docker import (CreateContainer, PullImage, StartContainer)
image = PullImage(
repository="georgeshsh/main",
tag="latest")
create_container = CreateContainer(
image_name="georgeshsh/main:latest")
start_container = StartContainer()
with Flow("docker-task-example",
run_config=UniversalRun(labels=["local"])
) as flow:
container_id = create_container(container_name="test_container")
started = start_container(container_id=container_id)
flow.run()k
Kevin Kho
06/22/2022, 3:12 PMAh ok. It was just passing None as the container name 🤦♂️
g
George Shishorin
06/22/2022, 3:12 PMMaybe add some defaults?
image name, f.e.
k
Kevin Kho
06/22/2022, 3:14 PMYou’re welcome to make a PR, or to assert it’s not None 🙂 . But this is the only time I’ve seen this so I’m not too worried