Mitchell Bregman
06/13/2022, 8:01 PMDbtShellTask
; locally works all as expected. when i deploy to prefect cloud, I am getting a dbt: command not found
.. requirements include dbt-redshift
. My flow storage is Docker. when i try to build the Docker image locally, inside the container I also get dbt: command not found
even though it is installed in my python environment packages. Has anyone run into this before?dbt_run = DbtShellTask(
environment=ENVIRONMENT,
profile_name="redshift",
dbt_kwargs={
"type": "redshift",
"host": REDSHIFT_HOST,
"user": REDSHIFT_USER,
"password": REDSHIFT_PASSWORD,
"port": REDSHIFT_PORT,
"database": REDSHIFT_DATABASE,
"schema": REDSHIFT_SCHEMA,
"threads": 4,
"client_session_keep_alive": False,
},
helper_script=f"cd {GIT_REPOSITORY}/redshift",
overwrite_profiles=True,
return_all=True,
log_stdout=True,
log_stderr=True,
)
with Flow(name="dbt-xxx") as flow:
repo = clone_repo()
logs = dbt_run(
task_args={"name": "xxx"},
command="dbt run --models xxx", upstream_tasks=[repo]
)
Matthias
06/13/2022, 8:23 PMdbt-redshift
package in the container? You can verify by running pip list
in a shell in the container…Mitchell Bregman
06/13/2022, 8:26 PM# pip list | grep dbt
dbt-core 1.1.0
dbt-extractor 0.4.1
dbt-postgres 1.1.0
dbt-redshift 1.1.0
Anna Geller
Mitchell Bregman
06/13/2022, 8:35 PM/usr/local/bin/python -m dbt
still command not foundLeon Kozlowski
06/13/2022, 8:35 PM/usr/local/bin/python: No module named dbt.__main__; 'dbt' is a package and cannot be directly executed
compgen -ac | grep dbt
Which returned nothing for dbt
Then we uninstalled all dbt related packages:
pip uninstall dbt-core dbt-extractor dbt-postgres dbt-redshift
After a reinstall
pip install dbt-core dbt-redshift
compgen -ac | grep dbt
And dbt --help
works, I think it might have to do with our Docker imageMatthias
06/13/2022, 9:28 PMLeon Kozlowski
06/13/2022, 9:30 PM