Channels
#prefect-community
Title
h
Henning Holgersen
09/01/2022, 5:52 PMHas anyone used Prefect together with Meltano? I am able to get it to work locally with an agent and a shell task, but I have not been able to build a docker image containing both the meltano project and the prefect agent - although Meltano has built-in functionality for containerisation.
👀 1
✅ 1
r
Ryan Peden
09/01/2022, 6:18 PMHi Henning, are you using Prefect 2.x?
If it runs locally, it should be possible to make it work inside a container. Are you running into any specific problems when you try to create the Docker image?
A few more things to consider:
• When running it locally, do you have Meltano and the Prefect agent installed in different virtual environments?
â—¦ If so, are you doing the same thing in your container?
â—¦ At a glance, it looks like Meltano and Prefect 2 use different versions of some of the same Python libraries, so that might cause issues if you are installing them both in the system Python environment in the container.
h
Henning Holgersen
09/01/2022, 6:24 PMHi Ryan, I tested this using Prefect 1. I did not document my failures properly, but they were some coarse attempts at building on the Prefect image with the contents of the meltano dockerfile, and the other way around.
Basing it on the prefect image seemed the most logical for me, but the
meltano installI could document some more in a day or two, if anyone is interesting in building on the stuff that didn’t work.
r
Ryan Peden
09/01/2022, 6:28 PMInformation about things that didn't work as expected is always helpful, as long as gathering the data won't take up too much of your time. 🙂
h
Henning Holgersen
09/01/2022, 6:54 PMThis is, approximately, the smallest reproducible example I can make right now: Running meltano install in my venv that has meltano (latest) + prefect 1.2.4 installed works nicely, but the following fails:
Dockerfile:
Copy code
FROM prefecthq/prefect:1.2.4-python3.10
RUN apt update && \
apt install -y python3-dev libpq-dev
RUN pip install meltano
WORKDIR /el
COPY meltano.yml .
RUN meltano install
WORKDIR / Copy code
=> ERROR [6/7] RUN meltano install 0.8s
------
> [6/7] RUN meltano install:
#11 0.717 Traceback (most recent call last):
#11 0.717 File "/usr/local/bin/meltano", line 5, in <module>
#11 0.717 from meltano.cli import main
#11 0.717 File "/usr/local/lib/python3.10/site-packages/meltano/cli/__init__.py", line 20, in <module>
#11 0.717 from . import ( # isort:skip # noqa: F401, WPS235
#11 0.717 File "/usr/local/lib/python3.10/site-packages/meltano/cli/add.py", line 8, in <module>
#11 0.717 from meltano.cli.params import pass_project
#11 0.717 File "/usr/local/lib/python3.10/site-packages/meltano/cli/params.py", line 11, in <module>
#11 0.717 from meltano.core.migration_service import MigrationError
#11 0.717 File "/usr/local/lib/python3.10/site-packages/meltano/core/migration_service.py", line 16, in <module>
#11 0.717 from meltano.api.models.security import Role, RolePermissions
#11 0.717 File "/usr/local/lib/python3.10/site-packages/meltano/api/models/__init__.py", line 3, in <module>
#11 0.717 from flask_sqlalchemy import SQLAlchemy
#11 0.717 File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/__init__.py", line 14, in <module>
#11 0.717 from flask import _app_ctx_stack, abort, current_app, request
#11 0.717 File "/usr/local/lib/python3.10/site-packages/flask/__init__.py", line 4, in <module>
#11 0.718 from . import json as json
#11 0.718 File "/usr/local/lib/python3.10/site-packages/flask/json/__init__.py", line 8, in <module>
#11 0.718 from ..globals import current_app
#11 0.718 File "/usr/local/lib/python3.10/site-packages/flask/globals.py", line 56, in <module>
#11 0.718 app_ctx: "AppContext" = LocalProxy( # type: ignore[assignment]
#11 0.718 TypeError: LocalProxy.__init__() got an unexpected keyword argument 'unbound_message'
------
executor failed running [/bin/sh -c meltano install]: exit code: 14 Views