https://prefect.io logo
Title
j

jpuris

02/06/2023, 12:43 PM
Heya! [Moved here from #prefect-getting-started] I'm struggling to find resources / reference for scheduling run container tasks with Prefect. The prefect docs on Running flows with Docker does not really fit my case as it relies on python code present, but in my case all I need it to do is run a pre-build image. Currently I have cron entry to do this
0 * * * * docker compose -f /some/path/docker-compose.yml run --rm my_service my_command
the docker-compose.yml
version: '3.8'
services:

  my_service:
    build: app/.
    env_file:
      - app/.env
    logging:
      driver: journald
How can I tell prefect to create a deployment to • run a container with a pre-defined image (I want to build it separately.. the docker-compose currently takes care of that) • load .env file into the container's envs • run specified
my_cmd
(entry-point's args) All I really need is some sort of reference material of building such deployment 🤷
1
I could do this with a Shell block, but then I need to install docker insider of the agent container to run a shell task, that runs container in docker 😄 Sounds awful.. I'm hoping I can do this more natively with e.g. Docker Container Block.
j

jpuris

02/06/2023, 3:48 PM
Indeed! I had taken a look at this just about earlier and it sort of does what I'm looking for.. I'll tinker with this and see, if I can make it do what docker-compose does for us right now 😄
Thank you @Timo Vink
Unfortunately, there is something broken with
prefect_docker
lib or I'm mis-using it.. The problem is that the library is sort of useless as it does not wait until the container ("task") completes or exits on it's own.. More info in How do we wait for container to exit on its own? #33 Also Their example does not do what it is trying to convey at all, for example the
command="echo 'hello world!' && sleep 60"
will simply output
21:37:54.109 | INFO    | Flow run 'glorious-jerboa' - hello world! && sleep 60
and not print "Hello World!" in the logs and then sleep for 60 seconds.. 🤷 -- How do you guys run "containers" as prefect tasks? Seems like it should be trivial thing to do for an orchestration tool and I'm at loss.. 😞
👀 1
I suspect that this lib is intended for whole different purpose than I'm trying to use it for.. i.e. Starting / Stopping docker containers.. that's it. What I'm looking for is for prefect to run a "container job" as part of a task as in, if you'd run a shell script, but encapsulated in a docker image.
a

Andrew Huang

02/07/2023, 12:41 AM
Also Their example does not do what it is trying to convey at all, for example the
command="echo 'hello world!' && sleep 60"
will simply output
21:37:54.109 | INFO    | Flow run 'glorious-jerboa' - hello world! && sleep 60
other than not sleeping properly, it did echo “hello world” as a log statement. to get both of the commands working properly, you can put dump it into a shell script like and run
bash test.sh
j

jpuris

02/12/2023, 10:31 AM
Solved! Here is the issue in case anyone is looking for an example https://github.com/PrefectHQ/prefect-docker/issues/33#issuecomment-1426994799 Thank you very much to everyone involved 💙