https://prefect.io logo
Channels
Powered by
m

Matthew Blau

12/28/2020, 7:41 PM
Hello all, I am attempting to run a docker container from within a prefect flow. Currently the file that I am trying to run lives within the file system as seen https://pastebin.com/NjuF5yFR here in my code. I have a volume that needs to be mounted for it to be able to locate the file, but I am unsure of the syntax. What am I doing wrong?
z

Zanie

12/28/2020, 7:47 PM
Hi @Matthew Blau! This is documented in the 
docker
create_container
function
Copy code
**Using volumes**

        Volume declaration is done in two parts. Provide a list of
        paths to use as mountpoints inside the container with the
        ``volumes`` parameter, and declare mappings from paths on the host
        in the ``host_config`` section.

        .. code-block:: python

            container_id = cli.create_container(
                'busybox', 'ls', volumes=['/mnt/vol1', '/mnt/vol2'],
                host_config=cli.create_host_config(binds={
                    '/home/user1/': {
                        'bind': '/mnt/vol2',
                        'mode': 'rw',
                    },
                    '/var/www': {
                        'bind': '/mnt/vol1',
                        'mode': 'ro',
                    }
                })
            )

        You can alternatively specify binds as a list. This code is equivalent
        to the example above:

        .. code-block:: python

            container_id = cli.create_container(
                'busybox', 'ls', volumes=['/mnt/vol1', '/mnt/vol2'],
                host_config=cli.create_host_config(binds=[
                    '/home/user1/:/mnt/vol2',
                    '/var/www:/mnt/vol1:ro',
                ])
            )
You’ll need to pass in the 
host_config
as well
m

Matthew Blau

12/28/2020, 7:56 PM
Hi Michael, so my line 14 needs to be a list and the volume is pointing to where the files live on the file system then, if I am understanding you correctly? As for the host_config, what are you referring to exactly. Forgive me, I am really new to workflow orchestration and Prefect and am trying to learn as much as I can with this.
z

Zanie

12/28/2020, 8:00 PM
I believe L14 can remain a single string but if it still doesn’t work changing it to a list may help. Specifying a volume just creates a reference to it in the docker container but you need to map the volume to a folder in the container, this is the “bind” which you do with a host config
The examples above show using 
create_host_config
to generate a formatted config for you, e.g. 
cli = docker.APIClient()
host_config = cli.create_host_config(binds=["/path/in/container:/volume/you/added"])
You can also just pass 
host_config={"Binds": ["/path/in/container:/volume/you/added"]}
m

Matthew Blau

12/28/2020, 8:07 PM
So I will need to add that bit of code into the flow I am creating, then? My docker-compose.yml has So my docker-compose.yml has " volumes: - .:/app" for the volumes information
z

Zanie

12/28/2020, 8:08 PM
You would add it to your 
CreateContainer
call
Copy code
container = CreateContainer(
    image_name="testphp_app:latest",
    volumes= "/home/mblau/projects/testphp",
    command= "php DatabaseTest.php"
)
docker-compose
doesn’t seem relevant to the code you posted
m

Matthew Blau

12/28/2020, 8:11 PM
I usually run the file with docker-compose and am now having prefect run it. the volume assignment in the create container call is not working as I am unable to open the input file DatabaseTest.php.
https://pastebin.com/wZ9RzEPm full error that I am currently getting
z

Zanie

12/28/2020, 8:43 PM
Did you add a host_config?
m

Matthew Blau

12/28/2020, 8:43 PM
Something like
Copy code
container = CreateContainer(
    image_name="testphp_app:latest",
    host_config={"Binds": "/home/mblau/projects/testphp/:/app"},
    volumes= "/home/mblau/projects/testphp/:/app",
    command= "php DatabaseTest.php"
)
yes?
z

Zanie

12/28/2020, 8:47 PM
Yeah then you’d want your command to be 
php /app/DatabaseTest.php
And the value for 
Binds
needs to be a list I think
(although I’m not certain what docker requires)
Perhaps if they’re both single strings it’s happy
m

Matthew Blau

12/28/2020, 9:05 PM
@Zanie Binds complains that it needs to be a string but it is at least no longer complaining about it not being able to open the file. Current error https://pastebin.com/NsV1mDAj
z

Zanie

12/28/2020, 9:06 PM
Ah sorry, we’re calling 
create_host_config
for you with the contents of 
host_config
You’ll want to use the
Copy code
binds=[
                    '/home/user1/:/mnt/vol2',
                    '/var/www:/mnt/vol1:ro',
                ]
syntax
m

Matthew Blau

12/28/2020, 9:07 PM
so Binds instead of host_config=
z

Zanie

12/28/2020, 9:07 PM
so 
host_config={"binds": ["/home/mblau/projects/testphp/:/app"]}
m

Matthew Blau

12/28/2020, 9:10 PM
Woo, well that runs it now. The issue I have now is with SQL Server login which is outside of Prefect's control, I would think so I don't think you can help any more. But thank you!
Have to verify it runs in the UI, of course. But the issue seems to be SQL server at this point
z

Zanie

12/28/2020, 9:12 PM
Glad it’s working 🙂
m

Matthew Blau

12/28/2020, 9:12 PM
I really do like Prefect. I find the docs much easier for a beginner to follow than Airflow. And I am thankful for this community!
So I would have to just pass in a list of files for more complicated containers, like ones with .env files and such, correct?
@Zanie one last question, I hope, but I can verify that via the CLI it works. But via the UI it does not seem to. Do I need to start a specific agent in order for this flow to execute in the UI?
z

Zanie

12/28/2020, 9:31 PM
A local agent should be fine
m

Matthew Blau

12/28/2020, 9:41 PM
Nice! Works perfectly. Got some more playing around with this before I can propose to my boss to deploy this corp wide but I like what we can do with it
z

Zanie

12/28/2020, 9:42 PM
Best of luck! Feel free to reach out to our sales team if you need help pitching it 😄
Powered by