https://prefect.io
Join Slack
Hi there - I am running into an issue with the `DbtShellTask` ; locally works all as expected. when ...
# ask-community
m
Hi there - I am running into an issue with the 
DbtShellTask
; locally works all as expected. when i deploy to prefect cloud, I am getting a 
dbt: command not found
.. requirements include 
dbt-redshift
. My flow storage is Docker. when i try to build the Docker image locally, inside the container I also get 
dbt: command not found
even though it is installed in my python environment packages. Has anyone run into this before?
âś… 1
đź‘€ 1
for context:
Copy code
dbt_run = DbtShellTask(
    environment=ENVIRONMENT,
    profile_name="redshift",
    dbt_kwargs={
        "type": "redshift",
        "host": REDSHIFT_HOST,
        "user": REDSHIFT_USER,
        "password": REDSHIFT_PASSWORD,
        "port": REDSHIFT_PORT,
        "database": REDSHIFT_DATABASE,
        "schema": REDSHIFT_SCHEMA,
        "threads": 4,
        "client_session_keep_alive": False,
    },
    helper_script=f"cd {GIT_REPOSITORY}/redshift",
    overwrite_profiles=True,
    return_all=True,
    log_stdout=True,
    log_stderr=True,
)
and then within flow:
Copy code
with Flow(name="dbt-xxx") as flow:
    repo = clone_repo()
    logs = dbt_run(
        task_args={"name": "xxx"},
        command="dbt run --models xxx", upstream_tasks=[repo]
    )
clone works just fine
m
Forgot to install 
dbt-redshift
package in the container? You can verify by running 
pip list
in a shell in the container…
upvote 1
m
its installed..
Copy code
# pip list | grep dbt
dbt-core                    1.1.0
dbt-extractor               0.4.1
dbt-postgres                1.1.0
dbt-redshift                1.1.0
a
Might be a PYTHONPATH issue, also if you run your docker container with sudo, it changes the directory and this may cause issues - worth checking your user in the container
m
even if i do something like 
/usr/local/bin/python -m dbt
still command not found
l
Running that command gives me:
Copy code
/usr/local/bin/python: No module named dbt.__main__; 'dbt' is a package and cannot be directly executed
(we’re working on the same issue)
Doing some more digging we checked for active commands that we could run using the following:
Copy code
compgen -ac | grep dbt
Which returned nothing for dbt Then we uninstalled all dbt related packages:
Copy code
pip uninstall dbt-core dbt-extractor dbt-postgres dbt-redshift
After a reinstall
Copy code
pip install dbt-core dbt-redshift
Afterwards, this shows dbt is available
Copy code
compgen -ac | grep dbt
And 
dbt --help
works, I think it might have to do with our Docker image
m
Strange… is it possible to share your Dockerfile?
l
I think we figured it out, we use a builder step in our dockerfile and copy site packages to final step, which was wiping out the CLI, after moving the dbt install in the final step its seems that the dbt CLI is available
đź‘Ť 2
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by