https://prefect.io logo
Powered by
Title
m

Mitchell Bregman

06/13/2022, 8:01 PM
Hi there - I am running into an issue with the 
DbtShellTask
; locally works all as expected. when i deploy to prefect cloud, I am getting a 
dbt: command not found
.. requirements include 
dbt-redshift
. My flow storage is Docker. when i try to build the Docker image locally, inside the container I also get 
dbt: command not found
even though it is installed in my python environment packages. Has anyone run into this before?
âś… 1
đź‘€ 1
for context: 
dbt_run = DbtShellTask(
    environment=ENVIRONMENT,
    profile_name="redshift",
    dbt_kwargs={
        "type": "redshift",
        "host": REDSHIFT_HOST,
        "user": REDSHIFT_USER,
        "password": REDSHIFT_PASSWORD,
        "port": REDSHIFT_PORT,
        "database": REDSHIFT_DATABASE,
        "schema": REDSHIFT_SCHEMA,
        "threads": 4,
        "client_session_keep_alive": False,
    },
    helper_script=f"cd {GIT_REPOSITORY}/redshift",
    overwrite_profiles=True,
    return_all=True,
    log_stdout=True,
    log_stderr=True,
)
and then within flow: 
with Flow(name="dbt-xxx") as flow:
    repo = clone_repo()
    logs = dbt_run(
        task_args={"name": "xxx"},
        command="dbt run --models xxx", upstream_tasks=[repo]
    )
clone works just fine
m

Matthias

06/13/2022, 8:23 PM
Forgot to install 
dbt-redshift
package in the container? You can verify by running 
pip list
in a shell in the container…
:upvote: 1
m

Mitchell Bregman

06/13/2022, 8:26 PM
its installed..
# pip list | grep dbt
dbt-core                    1.1.0
dbt-extractor               0.4.1
dbt-postgres                1.1.0
dbt-redshift                1.1.0
a

Anna Geller

06/13/2022, 8:33 PM
Might be a PYTHONPATH issue, also if you run your docker container with sudo, it changes the directory and this may cause issues - worth checking your user in the container
m

Mitchell Bregman

06/13/2022, 8:35 PM
even if i do something like 
/usr/local/bin/python -m dbt
still command not found
l

Leon Kozlowski

06/13/2022, 8:35 PM
Running that command gives me: 
/usr/local/bin/python: No module named dbt.__main__; 'dbt' is a package and cannot be directly executed
(we’re working on the same issue)
Doing some more digging we checked for active commands that we could run using the following: 
compgen -ac | grep dbt
Which returned nothing for dbt Then we uninstalled all dbt related packages: 
pip uninstall dbt-core dbt-extractor dbt-postgres dbt-redshift
After a reinstall 
pip install dbt-core dbt-redshift
Afterwards, this shows dbt is available 
compgen -ac | grep dbt
And 
dbt --help
works, I think it might have to do with our Docker image
m

Matthias

06/13/2022, 9:28 PM
Strange… is it possible to share your Dockerfile?
l

Leon Kozlowski

06/13/2022, 9:30 PM
I think we figured it out, we use a builder step in our dockerfile and copy site packages to final step, which was wiping out the CLI, after moving the dbt install in the final step its seems that the dbt CLI is available
đź‘Ť 2
2 Views
#prefect-communityJoin Slack
Powered by