https://prefect.io logo
#prefect-community
Title
# prefect-community
r

Rafal

06/15/2020, 4:46 PM
Unexpected error: ImageNotFound(HTTPError('404 Client Error: Not Found for url: <http+docker://localhost/v1.40/containers/create')>)
- this error occurs when I am trying to run my local docker image. I was trying to change docker_server_url but nothing worked. Error
Unexpected error: DockerException('Invalid bind address format: /var/run/docker.sock')
. Any ideas?
k

Kyle Moon-Wright

06/15/2020, 5:15 PM
Hey @Rafal! I will try to find more information for you. In the meantime, can you elaborate on what you’re trying to accomplish?
r

Rafal

06/15/2020, 5:17 PM
Sure! So I've got couple of containers which are doing different kind of tasks - downloading data, validation (using Great Expectations), transformation and loading into db. I would like to migrate solution from airflow and see how it will work on prefect. @Kyle Moon-Wright really thanks for help
k

Kyle Moon-Wright

06/15/2020, 5:33 PM
Nice! Is the image pull error from one of these converted Airflow tasks (as a Prefect flow)? Also, are you testing fully with the open-source Server UI?
r

Rafal

06/15/2020, 5:41 PM
Yes, I found mistake: /// 3xtimes slashes 😉 Seems like image is working. I am testing whole functionalities - including UI of course. And already have couple of things on mind: a) do I need to share the same volume of prefect container and docker task container (to store docker container task results) b) how to show container logs in UI - there is a task GetContainerLogs but argument is container_id ?! c) so, the question is how to retrive container_id on runtime, because only one method that I know needs container name but CreateContainer task does not allow to specify container name... I quess simple pipeline using containers with plugged volumes and showing logs in UI will be very helpful for community ;-)
k

Kyle Moon-Wright

06/15/2020, 7:49 PM
Hey sorry for the late reply, I’m glad you saw those slashes. You do not need to share the same volume, you should be able to store your results wherever you prefer or otherwise configure a volume for storage with your Agent. Additionally, you should be able to pass your container_id at runtime in the Flow definition like this:
Copy code
from prefect import Flow
from prefect.tasks.docker.containers import CreateContainer, StartContainer, GetContainerLogs

container = CreateContainer(
    image_name="<registry>/<imagename>:<tagname>"
)
start = StartContainer()
container_logs = GetContainerLogs()

with Flow("flow") as flow:
    start_container = start(container_id=container)
    code = container_logs(container_id=container, upstream_tasks=[start_container])

flow.run()
I agree this could be further explained in the docs and could be a great candidate for a Prefect idiom!
r

Rafal

06/16/2020, 5:38 PM
@Kyle Moon-Wright for sure I will try! And this idiom thing sounds interesting