m

    Matthew Blau

    1 year ago
    Hello all, I am attempting to run a docker container from within a prefect flow. Currently the file that I am trying to run lives within the file system as seen https://pastebin.com/NjuF5yFR here in my code. I have a volume that needs to be mounted for it to be able to locate the file, but I am unsure of the syntax. What am I doing wrong?
    Michael Adkins

    Michael Adkins

    1 year ago
    Hi @Matthew Blau! This is documented in the
    docker
    create_container
    function
    **Using volumes**
    
            Volume declaration is done in two parts. Provide a list of
            paths to use as mountpoints inside the container with the
            ``volumes`` parameter, and declare mappings from paths on the host
            in the ``host_config`` section.
    
            .. code-block:: python
    
                container_id = cli.create_container(
                    'busybox', 'ls', volumes=['/mnt/vol1', '/mnt/vol2'],
                    host_config=cli.create_host_config(binds={
                        '/home/user1/': {
                            'bind': '/mnt/vol2',
                            'mode': 'rw',
                        },
                        '/var/www': {
                            'bind': '/mnt/vol1',
                            'mode': 'ro',
                        }
                    })
                )
    
            You can alternatively specify binds as a list. This code is equivalent
            to the example above:
    
            .. code-block:: python
    
                container_id = cli.create_container(
                    'busybox', 'ls', volumes=['/mnt/vol1', '/mnt/vol2'],
                    host_config=cli.create_host_config(binds=[
                        '/home/user1/:/mnt/vol2',
                        '/var/www:/mnt/vol1:ro',
                    ])
                )
    You’ll need to pass in the
    host_config
    as well
    m

    Matthew Blau

    1 year ago
    Hi Michael, so my line 14 needs to be a list and the volume is pointing to where the files live on the file system then, if I am understanding you correctly? As for the host_config, what are you referring to exactly. Forgive me, I am really new to workflow orchestration and Prefect and am trying to learn as much as I can with this.
    Michael Adkins

    Michael Adkins

    1 year ago
    I believe L14 can remain a single string but if it still doesn’t work changing it to a list may help. Specifying a volume just creates a reference to it in the docker container but you need to map the volume to a folder in the container, this is the “bind” which you do with a host config
    The examples above show using
    create_host_config
    to generate a formatted config for you, e.g.
    cli = docker.APIClient()
    host_config = cli.create_host_config(binds=["/path/in/container:/volume/you/added"])
    You can also just pass
    host_config={"Binds": ["/path/in/container:/volume/you/added"]}
    m

    Matthew Blau

    1 year ago
    So I will need to add that bit of code into the flow I am creating, then? My docker-compose.yml has So my docker-compose.yml has " volumes: - .😕app" for the volumes information
    Michael Adkins

    Michael Adkins

    1 year ago
    You would add it to your
    CreateContainer
    call
    container = CreateContainer(
        image_name="testphp_app:latest",
        volumes= "/home/mblau/projects/testphp",
        command= "php DatabaseTest.php"
    )
    docker-compose
    doesn’t seem relevant to the code you posted
    m

    Matthew Blau

    1 year ago
    I usually run the file with docker-compose and am now having prefect run it. the volume assignment in the create container call is not working as I am unable to open the input file DatabaseTest.php.
    https://pastebin.com/wZ9RzEPm full error that I am currently getting
    Michael Adkins

    Michael Adkins

    1 year ago
    Did you add a host_config?
    m

    Matthew Blau

    1 year ago
    Something like
    container = CreateContainer(
        image_name="testphp_app:latest",
        host_config={"Binds": "/home/mblau/projects/testphp/:/app"},
        volumes= "/home/mblau/projects/testphp/:/app",
        command= "php DatabaseTest.php"
    )
    yes?
    Michael Adkins

    Michael Adkins

    1 year ago
    Yeah then you’d want your command to be
    php /app/DatabaseTest.php
    And the value for
    Binds
    needs to be a list I think
    (although I’m not certain what docker requires)
    Perhaps if they’re both single strings it’s happy
    m

    Matthew Blau

    1 year ago
    @Michael Adkins Binds complains that it needs to be a string but it is at least no longer complaining about it not being able to open the file. Current error https://pastebin.com/NsV1mDAj
    Michael Adkins

    Michael Adkins

    1 year ago
    Ah sorry, we’re calling
    create_host_config
    for you with the contents of
    host_config
    You’ll want to use the
    binds=[
                        '/home/user1/:/mnt/vol2',
                        '/var/www:/mnt/vol1:ro',
                    ]
    syntax
    m

    Matthew Blau

    1 year ago
    so Binds instead of host_config=
    Michael Adkins

    Michael Adkins

    1 year ago
    so
    host_config={"binds": ["/home/mblau/projects/testphp/:/app"]}
    m

    Matthew Blau

    1 year ago
    Woo, well that runs it now. The issue I have now is with SQL Server login which is outside of Prefect's control, I would think so I don't think you can help any more. But thank you!
    Have to verify it runs in the UI, of course. But the issue seems to be SQL server at this point
    Michael Adkins

    Michael Adkins

    1 year ago
    Glad it’s working 🙂
    m

    Matthew Blau

    1 year ago
    I really do like Prefect. I find the docs much easier for a beginner to follow than Airflow. And I am thankful for this community!
    So I would have to just pass in a list of files for more complicated containers, like ones with .env files and such, correct?
    @Michael Adkins one last question, I hope, but I can verify that via the CLI it works. But via the UI it does not seem to. Do I need to start a specific agent in order for this flow to execute in the UI?
    Michael Adkins

    Michael Adkins

    1 year ago
    A local agent should be fine
    m

    Matthew Blau

    1 year ago
    Nice! Works perfectly. Got some more playing around with this before I can propose to my boss to deploy this corp wide but I like what we can do with it
    Michael Adkins

    Michael Adkins

    1 year ago
    Best of luck! Feel free to reach out to our sales team if you need help pitching it 😄