https://prefect.io logo
Channels
Powered by
#prefect-community
Title
# prefect-community
m

Mitchell Bregman

06/13/2022, 8:01 PM
Hi there - I am running into an issue with the 
DbtShellTask
; locally works all as expected. when i deploy to prefect cloud, I am getting a 
dbt: command not found
.. requirements include 
dbt-redshift
. My flow storage is Docker. when i try to build the Docker image locally, inside the container I also get 
dbt: command not found
even though it is installed in my python environment packages. Has anyone run into this before?
âś… 1
đź‘€ 1
for context:
Copy code
dbt_run = DbtShellTask(
    environment=ENVIRONMENT,
    profile_name="redshift",
    dbt_kwargs={
        "type": "redshift",
        "host": REDSHIFT_HOST,
        "user": REDSHIFT_USER,
        "password": REDSHIFT_PASSWORD,
        "port": REDSHIFT_PORT,
        "database": REDSHIFT_DATABASE,
        "schema": REDSHIFT_SCHEMA,
        "threads": 4,
        "client_session_keep_alive": False,
    },
    helper_script=f"cd {GIT_REPOSITORY}/redshift",
    overwrite_profiles=True,
    return_all=True,
    log_stdout=True,
    log_stderr=True,
)
and then within flow:
Copy code
with Flow(name="dbt-xxx") as flow:
    repo = clone_repo()
    logs = dbt_run(
        task_args={"name": "xxx"},
        command="dbt run --models xxx", upstream_tasks=[repo]
    )
clone works just fine
m

Matthias

06/13/2022, 8:23 PM
Forgot to install 
dbt-redshift
package in the container? You can verify by running 
pip list
in a shell in the container…
upvote 1
m

Mitchell Bregman

06/13/2022, 8:26 PM
its installed..
Copy code
# pip list | grep dbt
dbt-core                    1.1.0
dbt-extractor               0.4.1
dbt-postgres                1.1.0
dbt-redshift                1.1.0
a

Anna Geller

06/13/2022, 8:33 PM
Might be a PYTHONPATH issue, also if you run your docker container with sudo, it changes the directory and this may cause issues - worth checking your user in the container
m

Mitchell Bregman

06/13/2022, 8:35 PM
even if i do something like 
/usr/local/bin/python -m dbt
still command not found
l

Leon Kozlowski

06/13/2022, 8:35 PM
Running that command gives me:
Copy code
/usr/local/bin/python: No module named dbt.__main__; 'dbt' is a package and cannot be directly executed
(we’re working on the same issue)
Doing some more digging we checked for active commands that we could run using the following:
Copy code
compgen -ac | grep dbt
Which returned nothing for dbt Then we uninstalled all dbt related packages:
Copy code
pip uninstall dbt-core dbt-extractor dbt-postgres dbt-redshift
After a reinstall
Copy code
pip install dbt-core dbt-redshift
Afterwards, this shows dbt is available
Copy code
compgen -ac | grep dbt
And 
dbt --help
works, I think it might have to do with our Docker image
m

Matthias

06/13/2022, 9:28 PM
Strange… is it possible to share your Dockerfile?
l

Leon Kozlowski

06/13/2022, 9:30 PM
I think we figured it out, we use a builder step in our dockerfile and copy site packages to final step, which was wiping out the CLI, after moving the dbt install in the final step its seems that the dbt CLI is available
đź‘Ť 2
2 Views
Powered by