Eric Mauser
07/27/2021, 8:52 PMflow.py
locally it returns that both tasks were successful, however I can see that it didn't actually run the dbt package. Any thoughts here? The dbt package is just in the same folder as the flow.py
. I'm assuming this all gets packaged up and run on ECS.Kevin Kho
07/27/2021, 8:53 PMEric Mauser
07/27/2021, 8:53 PMimport prefect
from prefect.tasks.dbt import DbtShellTask
from prefect.storage import Docker
from prefect.run_configs import ECSRun
from prefect import task, Flow, Parameter
STORAGE = Docker(registry_url='<myrole>',
image_name='dbt',
image_tag='latest')
RUN_CONFIG = ECSRun(execution_role_arn='<myrole>',
image = 'dbt',
labels=['dev'])
with Flow('dbt', storage=STORAGE, run_config=RUN_CONFIG) as flow:
task = DbtShellTask(profile_name='default', profiles_dir='')(command = 'dbt deps')
task2 = DbtShellTask(profile_name='default', profiles_dir='')(command = 'dbt run')
flow.register('<myproject>')
flow.run()
Kevin Kho
07/27/2021, 8:58 PMstream_output=True
and `return_all=True`as seen here so we can get the logs from the Shell Task?
Also, I guess you can try using the container built and running those commands without Prefect to see if they work?Eric Mauser
07/27/2021, 9:34 PMimage=fishtownanalytics/dbt:0.19.2
or do I need to build a custom image that combines prefect and dbt?Kevin Kho
07/27/2021, 9:38 PM