Joe Schueller
09/27/2022, 3:58 PMprefect-dbt
and can successfully execute a flow using the trigger_dbt_cli_command
task, but dbt doesn't actually do anything.Jeff Hale
09/27/2022, 5:54 PMJoe Schueller
09/27/2022, 6:23 PMfrom prefect import flow
from prefect_airbyte.connections import trigger_sync
from prefect_dbt.cli.commands import trigger_dbt_cli_command
@flow(name='Airbyte')
def run_airbyte_syncs():
airbyte_sync = trigger_sync (
airbyte_server_host='localhost',
airbyte_server_port=8000,
connection_id='5fdd8809-b7c6-4568-a6d2-72e17b648bb1',
poll_interval_s=3,
timeout=1800,
status_updates=True
)
return airbyte_sync
@flow(name='dbt')
def run_dbt_commands():
dbt_result = trigger_dbt_cli_command(command='dbt run',
profiles_dir='C:\\Users\\JSchueller\\.dbt',
project_dir='C:\\Users\\JSchueller\\etl-architecture-demo\\etl_architecture_demo',
shell='powershell',
return_all=True
)
return dbt_result
@flow(name='execute')
def execute():
# run_airbyte_syncs()
run_dbt_commands()
if __name__ == '__main__':
execute()
terminal logs the following:
10:37:02.302 | INFO | prefect.engine - Created flow run 'rose-zebu' for flow 'execute'
10:37:03.752 | INFO | Flow run 'rose-zebu' - Created subflow run 'icy-wallaby' for flow 'dbt'
10:37:03.936 | INFO | Flow run 'icy-wallaby' - Created task run 'trigger_dbt_cli_command-321ca940-0' for task 'trigger_dbt_cli_command'
10:37:03.937 | INFO | Flow run 'icy-wallaby' - Executing 'trigger_dbt_cli_command-321ca940-0' immediately...
10:37:04.036 | INFO | Task run 'trigger_dbt_cli_command-321ca940-0' - Running dbt command: dbt run --profiles-dir C:\Users\JSchueller\.dbt --project-dir C:\Users\JSchueller\etl-architecture-demo\etl_architecture_demo
10:37:09.409 | INFO | Task run 'trigger_dbt_cli_command-321ca940-0' - Finished in state Completed()
10:37:09.508 | INFO | Flow run 'icy-wallaby' - Finished in state Completed()
10:37:09.581 | INFO | Flow run 'rose-zebu' - Finished in state Completed('All states completed.')
If I run the same dbt command in pwsh, dbt actually picks it up and runs it.
~ prefect-env 3.9.12 dbt run --profiles-dir 'C:\Users\JSchueller\.dbt' --project-dir 'C:\Users\JSchueller\etl-architecture-demo\etl_architecture_demo'
14:25:04 Running with dbt=1.2.1
14:25:04 Unable to do partial parsing because of a dbt version mismatch. Saved manifest version: 1.1.2. Current version: 1.2.1.
14:25:05 Found 3 models, 11 tests, 0 snapshots, 0 analyses, 267 macros, 0 operations, 0 seed files, 2 sources, 0 exposures, 0 metrics
14:25:05
14:25:09 Concurrency: 1 threads (target='dev')
14:25:09
14:25:09 1 of 3 START view model dbt.stg_accounts ....................................... [RUN]
14:25:11 1 of 3 OK created view model dbt.stg_accounts .................................. [SUCCESS 1 in 1.29s]
14:25:11 2 of 3 START view model dbt.stg_contacts ....................................... [RUN]
14:25:12 2 of 3 OK created view model dbt.stg_contacts .................................. [SUCCESS 1 in 1.27s]
14:25:12 3 of 3 START table model dbt.dim_accounts ...................................... [RUN]
14:25:23 3 of 3 OK created table model dbt.dim_accounts ................................. [SUCCESS 1 in 11.48s]
14:25:23
14:25:23 Finished running 2 view models, 1 table model in 0 hours 0 minutes and 17.79 seconds (17.79s).
14:25:23
14:25:23 Completed successfully
14:25:23
14:25:23 Done. PASS=3 WARN=0 ERROR=0 SKIP=0 TOTAL=3
versions:
prefect: 2.4.2 (rebuilt my venv today, I was having this same problem running 2.3.2)
prefect-dbt: 0.2.2Jeff Hale
09/29/2022, 12:23 PMJoe Schueller
09/29/2022, 1:54 PMAndrew Huang
09/29/2022, 3:22 PMJoe Schueller
09/29/2022, 7:12 PMRuntimeWarning: coroutine 'shell_run_command' was never awaited
:
from prefect_shell.commands import *
c = "echo 'Hello world!'"
shell_run_command.fn(command=c, shell='powershell', return_all=True)
so I guess the event loop is closing before the command has a chance to run?Andrew Huang
09/29/2022, 7:13 PMshell_run_command.fn
without Prefect managing it is an async func; you may need to add
asyncio.run(shell_run_command.fn(...))
or wrap it in a Prefect flowJoe Schueller
09/29/2022, 7:14 PMAndrew Huang
09/29/2022, 7:15 PMJoe Schueller
09/29/2022, 7:17 PMshell_run_command
is trying to be opened. I imagine that's not supposed to happenAndrew Huang
09/29/2022, 7:22 PMecho "hello world"
is not working?import asyncio
from prefect_shell.commands import *
c = "echo 'Hello world!'"
asyncio.run(shell_run_command.fn(command=c, shell='powershell', return_all=True))
Joe Schueller
09/29/2022, 7:27 PMAndrew Huang
09/29/2022, 7:29 PMpowershell prefect-temporary.sh
from anyio import open_process
not sure how well supported that is
https://github.com/PrefectHQ/prefect-shell/blob/main/prefect_shell/commands.py#L75
another thing to check is if you set the log levels to debug
stream_level=logging.DEBUG
unfortunately, I don’t have access to a windows machine to test these out myselfJoe Schueller
09/29/2022, 7:39 PMprefect-temporary.sh
, do you mean specifically a bash script?Andrew Huang
09/29/2022, 7:40 PMwith tempfile.NamedTemporaryFile(prefix="prefect-") as tmp:
if helper_command:
tmp.write(helper_command.encode())
tmp.write(os.linesep.encode())
tmp.write(command.encode())
tmp.flush()
if sys.platform == "win32":
shell_command = " ".join(shell_command)
Joe Schueller
09/29/2022, 7:46 PMshell_run_command
to prefect-temporary
(whatever it winds up naming it), then executing as [shell] prefect-temporary
, correct?Andrew Huang
09/29/2022, 7:54 PMif shell.lower() == "powershell": suffix = ".ps1"
and pass suffix to `
NamedTemporaryFile(suffix=suffix)
Joe Schueller
09/29/2022, 7:56 PMAndrew Huang
09/29/2022, 7:57 PMMichael Holvey
09/30/2022, 12:18 AMtrigger_dbt_cli_command
taking a shell param in the docs?Joe Schueller
09/30/2022, 3:38 PMtrigger_dbt_cli_command
is a part of the prefect_dbt
library:
https://prefecthq.github.io/prefect-dbt/Michael Holvey
09/30/2022, 3:44 PMJoe Schueller
09/30/2022, 6:37 PMMichael Holvey
09/30/2022, 7:42 PMJoe Schueller
09/30/2022, 9:26 PMAndrew Huang
09/30/2022, 9:28 PMJoe Schueller
10/02/2022, 7:05 PMtest_run_shell_command_no_output
fail when using powershell because it handles things so differently. I'm trying to wrap my head around how to deal with this and I'm not getting far. any ideas spring to mind?Andrew Huang
10/03/2022, 7:53 PMJoe Schueller
10/06/2022, 9:22 PMAndrew Huang
10/07/2022, 10:52 PMMichael Holvey
10/07/2022, 10:55 PMAndrew Huang
10/07/2022, 11:50 PMpip install -U prefect-shell
)