https://prefect.io logo
Title
h

Hans Lellelid

02/07/2023, 6:01 PM
Hi folks -- I feel like I'm missing something, but is there support for running a docker container as a task in Prefect 2.x? I see https://docs-v1.prefect.io/api/0.15.13/tasks/docker.html which looks like what I want in 1.x, but so far the only Docker-related references I've found is how to run flows within a container. Specifically our use case is that there's a tool (not python) that someone has bundled into an image for portability. It does it's thing, produces some output (tbd on whether that's just to stdout or saving to files in the container that need to get copied out) and we'd like to put this into a prefect-managed pipeline. Edit: I do see the prefect-docker collection (https://prefecthq.github.io/prefect-docker/). This looks like what I want -- or at least pretty close. I'd like ability to pull off files (or maybe return standard out from the command being run?) ... I'll start looking through the source to see how I might add that functionality, since I don't see an obvious answer in the exposed API.
a

Andrew Huang

02/07/2023, 6:14 PM
h

Hans Lellelid

02/07/2023, 6:15 PM
Yes, I think you're right . I've been looking through that. I guess getting logs will return anything to stdout (or stderr?).
a

Andrew Huang

02/07/2023, 6:15 PM
probably both?
h

Hans Lellelid

02/07/2023, 6:16 PM
Yes, I think it's both. And to wait for the container to complete, I assume I probably pass detach=True ... trying to verify that now.
to wait for container, you can also use the wait method:
h

Hans Lellelid

02/07/2023, 6:17 PM
Lol, thank you! 🙂
It looks like I could also copy off files using docker-py API, would just want to copy the preject-docker pattern of wrapping these sync tasks.
👀 1
At the moment, you may hold onto the container and call the
wait
method.
Yeah, I saw this as well in the docker-py API, so I presume we could do something like:
await run_sync_in_worker_thread(container.wait)
@Andrew Huang, thanks for pointing me in the right directions here! It looks like it would be worth me making a PR to the prefect-docker to also support the idea of "docker run".
Lol, I see you pointed out the same thing:
I suppose the
run
method could be useful too in addition to create / start.
a

Andrew Huang

02/07/2023, 6:36 PM
Yeah we appreciate contributions! Thanks! Let me know if there’s anything I can help with
🙌 1
h

Hans Lellelid

02/07/2023, 6:36 PM
Yeah, it looks like we need this functionality, so I'm happy to make those additions and submit a PR.
:thank-you: 1