Has anyone used Prefect together with Meltano? I am able to get it to work locally with an agent and a shell task, but I have not been able to build a docker image containing both the meltano project and the prefect agent - although Meltano has built-in functionality for containerisation.

Hi Henning, are you using Prefect 2.x? If it runs locally, it should be possible to make it work inside a container. Are you running into any specific problems when you try to create the Docker image? A few more things to consider: • When running it locally, do you have Meltano and the Prefect agent installed in different virtual environments? ◦ If so, are you doing the same thing in your container? ◦ At a glance, it looks like Meltano and Prefect 2 use different versions of some of the same Python libraries, so that might cause issues if you are installing them both in the system Python environment in the container.

Hi Ryan, I tested this using Prefect 1. I did not document my failures properly, but they were some coarse attempts at building on the Prefect image with the contents of the meltano dockerfile, and the other way around. Basing it on the prefect image seemed the most logical for me, but the

meltano install

command failed when building on the prefect image.

Information about things that didn't work as expected is always helpful, as long as gathering the data won't take up too much of your time. 🙂

This is, approximately, the smallest reproducible example I can make right now: Running meltano install in my venv that has meltano (latest) + prefect 1.2.4 installed works nicely, but the following fails: Dockerfile:

FROM prefecthq/prefect:1.2.4-python3.10

RUN apt update && \
	apt install -y python3-dev libpq-dev

RUN pip install meltano

WORKDIR /el
COPY meltano.yml .

RUN meltano install

WORKDIR /

Error message:

=> ERROR [6/7] RUN meltano install                                                                                                                                                                                  0.8s 
------                                                                                                                                                                                                                    
 > [6/7] RUN meltano install:                                                                                                                                                                                             
#11 0.717 Traceback (most recent call last):                                                                                                                                                                              
#11 0.717   File "/usr/local/bin/meltano", line 5, in <module>                                                                                                                                                            
#11 0.717     from meltano.cli import main                                                                                                                                                                                
#11 0.717   File "/usr/local/lib/python3.10/site-packages/meltano/cli/__init__.py", line 20, in <module>                                                                                                                  
#11 0.717     from . import (  # isort:skip # noqa: F401, WPS235
#11 0.717   File "/usr/local/lib/python3.10/site-packages/meltano/cli/add.py", line 8, in <module>
#11 0.717     from meltano.cli.params import pass_project
#11 0.717   File "/usr/local/lib/python3.10/site-packages/meltano/cli/params.py", line 11, in <module>
#11 0.717     from meltano.core.migration_service import MigrationError
#11 0.717   File "/usr/local/lib/python3.10/site-packages/meltano/core/migration_service.py", line 16, in <module>
#11 0.717     from meltano.api.models.security import Role, RolePermissions
#11 0.717   File "/usr/local/lib/python3.10/site-packages/meltano/api/models/__init__.py", line 3, in <module>
#11 0.717     from flask_sqlalchemy import SQLAlchemy
#11 0.717   File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/__init__.py", line 14, in <module>
#11 0.717     from flask import _app_ctx_stack, abort, current_app, request
#11 0.717   File "/usr/local/lib/python3.10/site-packages/flask/__init__.py", line 4, in <module>
#11 0.718     from . import json as json
#11 0.718   File "/usr/local/lib/python3.10/site-packages/flask/json/__init__.py", line 8, in <module>
#11 0.718     from ..globals import current_app
#11 0.718   File "/usr/local/lib/python3.10/site-packages/flask/globals.py", line 56, in <module>
#11 0.718     app_ctx: "AppContext" = LocalProxy(  # type: ignore[assignment]
#11 0.718 TypeError: LocalProxy.__init__() got an unexpected keyword argument 'unbound_message'
------
executor failed running [/bin/sh -c meltano install]: exit code: 1