https://prefect.io logo
Channels
Powered by
#prefect-server
Title
# prefect-server
e

Eric Mauser

07/27/2021, 8:52 PM
I'm trying to use ECSRun to run a couple of DBTShellTasks. When I run this 
flow.py
locally it returns that both tasks were successful, however I can see that it didn't actually run the dbt package. Any thoughts here? The dbt package is just in the same folder as the 
flow.py
. I'm assuming this all gets packaged up and run on ECS.
k

Kevin Kho

07/27/2021, 8:53 PM
Hey @Eric Mauser, what Storage did you use?
e

Eric Mauser

07/27/2021, 8:53 PM
Copy code
import prefect
from prefect.tasks.dbt import DbtShellTask
from prefect.storage import Docker
from prefect.run_configs import ECSRun
from prefect import task, Flow, Parameter

STORAGE = Docker(registry_url='<myrole>',
                 image_name='dbt',
                 image_tag='latest')

RUN_CONFIG = ECSRun(execution_role_arn='<myrole>',
                    image = 'dbt',
                    labels=['dev'])

with Flow('dbt', storage=STORAGE, run_config=RUN_CONFIG) as flow:
    task = DbtShellTask(profile_name='default', profiles_dir='')(command = 'dbt deps')
    task2 = DbtShellTask(profile_name='default', profiles_dir='')(command = 'dbt run')
flow.register('<myproject>')
flow.run()
@Kevin Kho Docker storage
k

Kevin Kho

07/27/2021, 8:58 PM
Docker Storage looks good and it seems like it would error out if it did not run in the right directory. I think we would try to need to get more logs here. Try 
stream_output=True
and `return_all=True`as seen here so we can get the logs from the Shell Task? Also, I guess you can try using the container built and running those commands without Prefect to see if they work?
e

Eric Mauser

07/27/2021, 9:34 PM
okay, looks like it is running locally now. However, it says dbt command not found when run on ECS. I'm guessing this is because I didn't sprecify an image so it is using the default prefect image. Can I just do 
image=fishtownanalytics/dbt:0.19.2
or do I need to build a custom image that combines prefect and dbt?
k

Kevin Kho

07/27/2021, 9:38 PM
I believe as long as the image has Python, prefect will install itself onto the container using additional commands (need pip also). If that doesn’t work, then I guess you have to create an image that contains both.
I tried digging through the code but not seeing it self-install so really not sure that’ll work.
3 Views
Powered by