not sure if this is the best channel, but my issue...
# prefect-dbt
d
not sure if this is the best channel, but my issue seems specific to how prefect + dbt are interacting sooo .. here it goes. i've got a bunch of flows setup with agent work pools. prefect triggers airbyte, and after a successful sync, a series of dbt jobs get triggered. both prefect and dbt are running in the same ec2 instance, but each has their own venv managed by pyenv in their respective folders. this works fine. when trying to move to a process-typed work pool, im getting errors with the dbt portion of my flows. airbyte is triggering just fine, but dbt refuses to run like it did before. ive compared debug logs between the agent-based pool and the worker-based pool, and i cant figure out where the difference is creeping in. the error code suggests python can't find the relevant libraries to get everything to play nice. nothing has changed about the flow code. any thoughts?
Copy code
Importing flow code from 'flows/testing/test-flow-dbt-deps.py:run_dbt_deps'
10:07:11 AM
prefect.flow_runs
Starting 'ConcurrentTaskRunner'; submitted tasks will be run concurrently...
10:07:11 AM
prefect.flow_runs
Executing flow 'run-dbt-deps' for flow run 'versatile-seriema'...
10:07:11 AM
prefect.flow_runs
Beginning execution...
10:07:11 AM
prefect.flow_runs
Writing the following commands to '/tmp/prefect-igdjymm3.sh':
dbt deps --profiles-dir /home/ec2-user/.dbt --project-dir /home/ec2-user/data-dbt-projects
10:07:11 AM
prefect.flow_runs
PID 31482 triggered with 1 commands running inside the PosixPath('/home/ec2-user/data-dbt-projects') directory.
10:07:11 AM
prefect.flow_runs
Waiting for PID 31482 to complete.
10:07:11 AM
prefect.flow_runs
Encountered exception during execution:
Traceback (most recent call last):
  File "/home/ec2-user/.pyenv/versions/venv_3-11-3_prefect/lib/python3.11/site-packages/prefect/engine.py", line 841, in orchestrate_flow_run
    result = await flow_call.aresult()
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/.pyenv/versions/venv_3-11-3_prefect/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 293, in aresult
    return await asyncio.wrap_future(self.future)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/.pyenv/versions/venv_3-11-3_prefect/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 318, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpb9mm9vjuprefect/data-prefect-main/flows/testing/test-flow-dbt-deps.py", line 17, in run_dbt_deps
    ).run()
      ^^^^^
  File "/home/ec2-user/.pyenv/versions/venv_3-11-3_prefect/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
    return call()
           ^^^^^^
  File "/home/ec2-user/.pyenv/versions/venv_3-11-3_prefect/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 398, in __call__
    return self.result()
           ^^^^^^^^^^^^^
  File "/home/ec2-user/.pyenv/versions/venv_3-11-3_prefect/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 284, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/.pyenv/versions/venv_3-11-3_prefect/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 168, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/.pyenv/versions/3.11.3/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/ec2-user/.pyenv/versions/venv_3-11-3_prefect/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 355, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/home/ec2-user/.pyenv/versions/venv_3-11-3_prefect/lib/python3.11/site-packages/prefect_shell/commands.py", line 396, in run
    await shell_process.wait_for_completion()
  File "/home/ec2-user/.pyenv/versions/venv_3-11-3_prefect/lib/python3.11/site-packages/prefect_shell/commands.py", line 177, in wait_for_completion
    raise RuntimeError(
RuntimeError: PID 31482 failed with return code 127.
10:07:11 AM
prefect.flow_runs
Finished in state Failed('Flow run encountered an exception. RuntimeError: PID 31482 failed with return code 127.')
streaming output from dbt, definitely seems like it's an issue with venvs...
in case anyone runs across this in the future, this was caused by a change in how i was managing the worker. with the old model (agent) i used supervisord. with the new model (worker) i was using systemd. tl,dr: systemd imputes a default path that didnt include any of venv management that comes along with pyenv. i decided to just re-use supervisord to manage the worker process and all is well.