https://prefect.io logo
Join SlackCommunities
Powered by
Hi all, I keep getting this error after specifying...
# ask-community
m
Hi all, I keep getting this error after specifying the profiles_dir in the DbtShellTask- it is the directory where the file is located so I'm not sure why it's not able to find it
Copy code
fatal: Not a dbt project (or any of the parent directories). Missing dbt_project.yml file
k
Hey @Madison Schott, I haven’t used dbt myself but have you seen the dbt guide to debug this here ? Could be that the DbtShellTask is not running in the directory that you expect. I assume you have the dbt_project.yml file?
m
the same dbt command runs fine from that directory locally
so I don't think the issue is with how the dbt model is set up
k
What is your Prefect setup? What storage, and run config are you using? Could you share the Prefect code?
m
Copy code
with Flow(name="dbt_flow") as f:
        task = DbtShellTask(
            profile_name='w_dev',
            log_stderr=True,
            environment='dev',
            dbt_kwargs = {
                "type": "snowflake",
                "database": x,
                "warehouse": x,
                "schema": x,
                "threads": 200
            },
            profiles_dir='/Users/madison/.dbt'
        )(command='dbt run -m tag:daily')
what do you mean by storage and run config?
k
Gotcha, are you using the Local Agent then? Default is Local Storage, so this is serialized and saved on your machine. Storage can also be Github/S3/Docker. Run Config is for how the flow is run (Local/Docker/Kubernetes). In this case there is a default Universal. But no need to pay attention to these for this. At what path are you starting your agent?
Actually seems right since you have absolute path. Looking at the DbtShellTask
Do you have both a 
profiles.yml
and 
dbt_project.yml
in that folder? Read a bit and checking that my understanding is right
m
ah yes all locally
I'm starting it in a separate directory I created for prefect
so it is a different directory than the prefect project file but that's why I specified the directory
k
Copying the dbt starter project and I’ll look into this
m
great, thanks!
k
I set up the tutorial project and everything but dbt errors for the Apple M1 so I get an error unfortunately. 
dbt run
doesn’t work. But I think I understand it now. 
profiles.yml
is in the 
profiles_dir
, but 
dbt run
looks for the 
dbt_project.yml
in the directory where it runs. So you have 
dbt run
running in the actual project folder, and it looks for the 
profiles.yml
in the 
profiles_dir
.
I believe the error you are seeing is because you specified the correct 
profiles_dir
, but 
dbt run
is not running where you expect it to be. There are two ways I think you can fix this. First, is run the Prefect Agent in the dbt project directory that contains the 
dbt_project.yml
. It will run the 
dbt run
on this folder.
The second thing is you can profile a 
helper script
to navigate to the correct directory. You can try 
helper_script = "cd /abspath/to/folder"
, and point it to the folder that contains the 
dbt_project.yml
m
that worked, thanks!
👍 1
c
Is there any way to log the output to the console ? I can run the DbtShellTask, but can't see whats going on, except that it succeeded. Have tried setting 
return_all
and 
log_stderr
to 
True
, but still not seeing any output.
Copy code
[2021-07-16 04:30:25+0000] INFO - prefect.FlowRunner | Beginning Flow run for 'dbt_flow'
[2021-07-16 04:30:25+0000] INFO - prefect.TaskRunner | Task 'DbtShellTask': Starting task run...
[2021-07-16 04:30:54+0000] INFO - prefect.TaskRunner | Task 'DbtShellTask': Finished task run for task with final state: 'Success'
[2021-07-16 04:30:54+0000] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
k
Are you on Windows @CA Lee?
c
I am using WSL 2.0 (Ubuntu 20.04) on Windows, developing in a vscode devcontainer
k
Try 
stream_output=True
✅ 1
c
Works great, thanks ! Could I also check how to add retries to 
DbtShellTask
? From the docs:
Copy code
@task(max_retries=3, retry_delay=timedelta(seconds=10))
However, if I try to wrap the shell task using the Prefect task decorator, I get the below error:
Copy code
ValueError: Could not infer an active Flow context while creating edge to <Task: DbtShellTask>. This often means you called a task outside a `with Flow(...)` block. If you're trying to run this task outside of a Flow context, you need to call `DbtShellTask(...).run(...)`
This was the code I attempted:
Copy code
from datetime import timedelta
from prefect import task, Flow
from prefect.tasks.dbt import DbtShellTask

FLOW_NAME = "[daily] [2300] dbt run"

@task(max_retries=3, retry_delay=timedelta(seconds=10))
def dbt_run():

    DbtShellTask(
        profile_name="jinlee",
        environment="dev",
        profiles_dir="/home/vscode/.dbt",
        helper_script="cd /workspaces/flowstate/transform",
        return_all=True,
        log_stderr=True,
        stream_output=True
    )(command='dbt run')


with Flow(
    FLOW_NAME,
) as flow:
    dbt_run()    

flow.run()
Appending 
.run()
behind the call to 
DbtShellTask
didn't work as well.
k
All tasks in the task library take in the 
kwargs
that the 
@task
decorator does so I think you can do
Copy code
DbtShellTask(
        profile_name="jinlee",
        environment="dev",
        profiles_dir="/home/vscode/.dbt",
        helper_script="cd /workspaces/flowstate/transform",
        return_all=True,
        log_stderr=True,
        stream_output=True,
        max_retries=3,
        ....
    )(command='dbt run')
Tasks can be called in other tasks using the 
.run()
so it would be
Copy code
@task(max_retries=3, retry_delay=timedelta(seconds=10))
def dbt_run():
    DbtShellTask(
        profile_name="jinlee",
        environment="dev",
        profiles_dir="/home/vscode/.dbt",
        helper_script="cd /workspaces/flowstate/transform",
        return_all=True,
        log_stderr=True,
        stream_output=True
    ).run(command='dbt run')
with Flow(
    FLOW_NAME,
) as flow:
    dbt_run()    
flow.run()
Open in Slack
PreviousNext
Powered by