Hi I am new to Prefect and am trying to get DBT flows up and Prefect Community #prefect-dbt

Hi! I am new to Prefect and am trying to get DBT f...

Tom A

06/08/2023, 2:59 PM

Hi! I am new to Prefect and am trying to get DBT flows up and running in my local prefect `2.10.12`/Python

3.10

environment and am hitting some errors. Is there anyone who can take a look and let me know if I'm doing something wrong? Going to put some more info in the thread to not clog up the channel..

Tom A

06/08/2023, 2:59 PM

This is my flow:

Copy code

from prefect import flow
from prefect_dbt.cli import DbtCliProfile, DbtCoreOperation

@flow
def trigger_dbt_flow() -> str:
    result = DbtCoreOperation(
        commands=["dbt debug", "dbt test"],
        project_dir="dbt-repo",
        profiles_dir="dbt-profiles"
    ).run()
    return result

trigger_dbt_flow()

Here is my directory structure:

Copy code

C:.
│   .prefectignore
│
└───flows
    │   test.py
    │
    ├───dbt-profiles
    │       profiles.yml
    │
    └───dbt-repo
		│   dbt_project.yml
        ├───analyses
        ├───macros   
        ├───models
        ├───seeds
        ├───snapshots
        └───tests

And here is the error:

Copy code

(prefect_env) PS C:\repos\prefect-poc\client\app\flows> python .\test.py
10:56:07.315 | INFO    | prefect.engine - Created flow run 'whimsical-dalmatian' for flow 'trigger-dbt-flow'
10:56:07.319 | INFO    | Flow run 'whimsical-dalmatian' - View at <http://172.31.105.50:4200/flow-runs/flow-run/8e0ee6bf-75bb-4208-a527-a37b694ac7da>
10:56:07.442 | INFO    | Flow run 'whimsical-dalmatian' - PID 44284 triggered with 2 commands running inside the '.' directory.
10:56:08.753 | INFO    | Flow run 'whimsical-dalmatian' - PID 44284 stream output:
Traceback (most recent call last):
  File "C:\repos\prefect-poc\client\app\flows\dbt.py", line 1, in <module>
10:56:08.754 | INFO    | Flow run 'whimsical-dalmatian' - PID 44284 stream output:
    from prefect import flow, task
ModuleNotFoundError: No module named 'prefect'
10:56:08.838 | INFO    | Flow run 'whimsical-dalmatian' - PID 44284 stream output:
Traceback (most recent call last):
  File "C:\repos\prefect-poc\client\app\flows\dbt.py", line 1, in <module>
10:56:08.841 | INFO    | Flow run 'whimsical-dalmatian' - PID 44284 stream output:
    from prefect import flow, task
ModuleNotFoundError: No module named 'prefect'
10:56:08.868 | ERROR   | Flow run 'whimsical-dalmatian' - Encountered exception during execution:
Traceback (most recent call last):
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\engine.py", line 750, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 181, in aresult
    return await asyncio.wrap_future(self.future)
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 194, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "C:\repos\prefect-poc\client\app\flows\test.py", line 10, in trigger_dbt_flow
    ).run()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\utilities\asyncutils.py", line 260, in coroutine_wrapper
    return call()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 245, in __call__
    return self.result()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 173, in result
    return self.future.result(timeout=timeout)
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\concurrent\futures\_base.py", line 438, in result
    return self.__get_result()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\concurrent\futures\_base.py", line 390, in __get_result
    raise self._exception
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 218, in _run_async
    result = await coro
  File "C:\Users\tanderson\AppData\Roaming\Python\Python310\site-packages\prefect_shell\commands.py", line 396, in run
    await shell_process.wait_for_completion()
  File "C:\Users\tanderson\AppData\Roaming\Python\Python310\site-packages\prefect_shell\commands.py", line 177, in wait_for_completion
    raise RuntimeError(
RuntimeError: PID 44284 failed with return code 1.
10:56:08.911 | ERROR   | Flow run 'whimsical-dalmatian' - Finished in state Failed('Flow run encountered an exception. RuntimeError: PID 44284 failed with return code 1.\n')
Traceback (most recent call last):
  File "C:\repos\prefect-poc\client\app\flows\test.py", line 13, in <module>
    trigger_dbt_flow()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\flows.py", line 511, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\engine.py", line 206, in enter_flow_run_engine_from_flow_call
    retval = from_sync.wait_for_call_in_loop_thread(
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\api.py", line 232, in wait_for_call_in_loop_thread
    return call.result()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 173, in result
    return self.future.result(timeout=timeout)
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\concurrent\futures\_base.py", line 438, in result
    return self.__get_result()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\concurrent\futures\_base.py", line 390, in __get_result
    raise self._exception
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 218, in _run_async
    result = await coro
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\client\utilities.py", line 40, in with_injected_client
    return await fn(*args, **kwargs)
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\engine.py", line 300, in create_then_begin_flow_run
    return await state.result(fetch=True)
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\states.py", line 91, in _get_state_result
    raise await get_state_exception(state)
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\engine.py", line 750, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 181, in aresult
    return await asyncio.wrap_future(self.future)
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 194, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "C:\repos\prefect-poc\client\app\flows\test.py", line 10, in trigger_dbt_flow
    ).run()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\utilities\asyncutils.py", line 260, in coroutine_wrapper
    return call()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 245, in __call__
    return self.result()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 173, in result
    return self.future.result(timeout=timeout)
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\concurrent\futures\_base.py", line 438, in result
    return self.__get_result()
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\concurrent\futures\_base.py", line 390, in __get_result
    raise self._exception
  File "C:\Users\tanderson\.conda\envs\prefect_env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 218, in _run_async
    result = await coro
  File "C:\Users\tanderson\AppData\Roaming\Python\Python310\site-packages\prefect_shell\commands.py", line 396, in run
    await shell_process.wait_for_completion()
  File "C:\Users\tanderson\AppData\Roaming\Python\Python310\site-packages\prefect_shell\commands.py", line 177, in wait_for_completion
    raise RuntimeError(
RuntimeError: PID 44284 failed with return code 1.

Sean Williams

06/08/2023, 3:18 PM

Copy code

ModuleNotFoundError: No module named 'prefect'

I think you might need to install prefect

pip install -U prefect

Tom A

06/08/2023, 3:21 PM

Hey Sean, thanks for taking a look, in the conda environment I'm trying to run this, prefect is installed ->

Copy code

(prefect_env) PS C:\repos\prefect-poc\client\app\flows> pip list
Package                    Version
-------------------------- --------
agate                      1.7.0
aiosqlite                  0.19.0
alembic                    1.11.1
anyio                      3.7.0
apprise                    1.4.0
argon2-cffi                21.3.0
argon2-cffi-bindings       21.2.0
asgi-lifespan              2.1.0
asn1crypto                 1.5.1
asttokens                  2.0.5
asyncpg                    0.27.0
attrs                      21.4.0
autopep8                   1.6.0
Babel                      2.12.1
backcall                   0.2.0
beautifulsoup4             4.10.0
bleach                     4.1.0
cachetools                 5.1.0
certifi                    2023.5.7
cffi                       1.15.1
charset-normalizer         3.1.0
click                      8.1.3
cloudpickle                2.2.1
colorama                   0.4.4
coolname                   2.2.0
croniter                   1.3.15
cryptography               40.0.2
dateparser                 1.1.8
dbt-core                   1.5.1
dbt-extractor              0.4.1
dbt-snowflake              1.5.1
debugpy                    1.5.1
decorator                  5.1.1
defusedxml                 0.7.1
docker                     6.1.3
entrypoints                0.4
exceptiongroup             1.1.1
executing                  0.8.3
fastapi                    0.96.0
filelock                   3.12.0
fsspec                     2023.5.0
future                     0.18.3
google-auth                2.6.6
google-auth-oauthlib       0.5.1
graphql-core               3.2.3
greenlet                   2.0.2
griffe                     0.29.0
gspread                    5.3.2
h11                        0.14.0
h2                         4.1.0
hologram                   0.0.16
hpack                      4.0.0
httpcore                   0.17.2
httpx                      0.24.1
hyperframe                 6.0.1
idna                       3.4
importlib-metadata         6.6.0
ipykernel                  6.9.2
ipython                    8.1.1
ipython-genutils           0.2.0
isodate                    0.6.1
jaraco.classes             3.2.3
jedi                       0.18.1
Jinja2                     3.1.2
jsonpatch                  1.32
jsonpointer                2.3
jsonschema                 4.4.0
jupyter-client             7.1.2
jupyter-core               4.9.2
jupyterlab-pygments        0.1.2
keyring                    23.13.1
kubernetes                 26.1.0
leather                    0.3.4
Logbook                    1.5.3
Mako                       1.2.4
Markdown                   3.4.3
markdown-it-py             2.2.0
MarkupSafe                 2.1.3
mashumaro                  3.6
matplotlib-inline          0.1.3
mdurl                      0.1.2
minimal-snowplow-tracker   0.0.2
mistune                    0.8.4
more-itertools             9.1.0
msgpack                    1.0.5
nbclient                   0.5.13
nbconvert                  6.4.4
nbformat                   5.2.0
nest-asyncio               1.5.4
networkx                   2.8.8
notebook                   6.4.9
oauthlib                   3.2.0
orjson                     3.9.0
oscrypto                   1.3.0
packaging                  21.3
pandocfilters              1.5.0
parsedatetime              2.4
parso                      0.8.3
pathspec                   0.11.1
pendulum                   2.1.2
pickleshare                0.7.5
pip                        23.0.1
prefect                    2.10.12
prefect-airbyte            0.2.0
prefect-dbt                0.3.1
prefect-shell              0.1.5
prefect-snowflake          0.26.1
prometheus-client          0.13.1
prompt-toolkit             3.0.28
protobuf                   4.23.2
psutil                     5.9.0
pure-eval                  0.2.2
pyasn1                     0.4.8
pyasn1-modules             0.2.8
pycodestyle                2.8.0
pycparser                  2.21
pycryptodomex              3.18.0
pydantic                   1.10.9
Pygments                   2.15.1
PyJWT                      2.7.0
pyOpenSSL                  23.2.0
pyparsing                  3.0.7
pyrsistent                 0.18.1
python-dateutil            2.8.2
python-slugify             8.0.1
pytimeparse                1.1.8
pytz                       2023.3
pytzdata                   2020.1
pywin32                    306
pywin32-ctypes             0.2.0
pywinpty                   2.0.5
PyYAML                     6.0
pyzmq                      22.3.0
readchar                   4.0.5
regex                      2023.6.3
requests                   2.31.0
requests-oauthlib          1.3.1
rich                       13.4.1
rsa                        4.8
Send2Trash                 1.8.0
setuptools                 67.8.0
sgqlc                      16.2
six                        1.16.0
sniffio                    1.3.0
snowflake-connector-python 3.0.4
sortedcontainers           2.4.0
soupsieve                  2.3.1
SQLAlchemy                 2.0.15
sqlparse                   0.4.3
stack-data                 0.2.0
starlette                  0.27.0
terminado                  0.13.3
testpath                   0.6.0
text-unidecode             1.3
toml                       0.10.2
tornado                    6.1
traitlets                  5.1.1
typer                      0.9.0
typing_extensions          4.6.3
tzdata                     2023.3
tzlocal                    5.0.1
urllib3                    1.26.16
uvicorn                    0.22.0
wcwidth                    0.2.5
webencodings               0.5.1
websocket-client           1.5.2
websockets                 11.0.3
Werkzeug                   2.3.5
wheel                      0.38.4
zipp                       3.15.0

Do I need to do something special to tell it to run in the environment the flow run originated from?

Sean Williams

06/08/2023, 3:38 PM

I'm guessing it's something around the virtual environment. What do

which python

and

which -a pip

show?

Tom A

06/08/2023, 3:48 PM

I'm on windows, but I think this is similar:

Copy code

(prefect_env) PS C:\repos\prefect-poc\client\app\flows> python
Python 3.10.0 | packaged by conda-forge | (default, Nov 10 2021, 13:20:59) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.which('python')
'C:\\Users\\tanderson\\.conda\\envs\\prefect_env\\python.EXE'
>>> shutil.which('pip')    
'C:\\Users\\tanderson\\.conda\\envs\\prefect_env\\Scripts\\pip.EXE'

Tom A

06/09/2023, 8:26 PM

Just circling back, there was something wacky going on with conda, I swapped to a python venv and got it working!

Sean Williams

06/09/2023, 10:17 PM

Interesting. Thanks for circling back

10 Views

Open in Slack

Previous Next