https://prefect.io
Join Slack
Has anyone else ever followed this tutorial when using DbtShellTask? I've created my Dockerfile to b...
# prefect-server
m
Has anyone else ever followed this tutorial when using DbtShellTask? I've created my Dockerfile to be similar but am having issues with my dbt files https://www.lejimmy.com/distributed-data-pipelines-with-aws-ecs-fargate-and-prefect-cloud/
n
I’m running dbt with Prefect on ECS Fargate but not using Docker storage. Is dbt installed in your image that you run the flow in? What’s the issue with the dbt files? Are they included in your image as well or are you downloading them from somewhere externally when executing the flow?
👍 1
m
Yeah dbt is installed but it’s saying the directory that I’m copying is not a dbt project even though I have my dockerfile inside the project
I’m trying to copy them from the directory I have the dockerfile stored in- not sure if that is right though- my flow which created the docket image is in another file
n
Can you show some examples? It sounds like something is just off with how you define things in the Dockerfile, or use your built image
m
Copy code
# specify a base image
FROM prefecthq/prefect:latest

WORKDIR dbt_snowflake/

COPY . .

# install all dependencies
RUN pip install -U pip
RUN pip install dbt==0.17.2
that's my Dockerfile and this is what my folder structure looks like
Copy code
STORAGE = Docker(registry_url='xx',
                 image_name='prefect-flows',
                 dockerfile='/Users/madisonschott/dbt_snowflake/DockerFile')
that's in the file for my flow when I register it
but then also my DbtShellTask looks like this so I'm not sure how to handle that when trying to use the Docker image
Copy code
user_profile_dbt_run = dbt_task(helper_script = "cd /Users/madisonschott/dbt_snowflake",
                             command='dbt run')
n
If you build your image in the root of 
dbt_snowflake
the path that you cd into shouldn’t have your local mac’s path. I think you should 
"cd dbt_snowflake"
You could check locally where your dbt project is within the image with 
docker run --rm -it <imagename> sh
, that way you should see where you need to cd
I also had to juggle with the profiles a bit. This is my code
Copy code
DBT_REPO_PATH = "/tmp/dbt"

dbt = DbtShellTask(
    helper_script=f"cd {DBT_REPO_PATH}",
    log_stderr=True,
    log_stdout=True,
    profiles_dir=DBT_REPO_PATH,
    stream_output=True,
    checkpoint=False,
)
Then I run it in the flow with 
dbt(command="dbt run")
m
If I'm using 
dbt_kwargs
do I even need to specify the profiles_dir?
Does the Dockerfile look right to you? For how my folder structure is?
n
Not sure, can’t remember the turns I took when testing myself. Try it without first
Yes I think the Dockerfile is OK. Your fix will probably be on the flow side of things. I’m not sure of the details when the image is built by Prefect and not yourself. That’s why I’d suggest looking inside the docker image to see where your dbt project is located in that file tree
j
If it helps, I am also using dbt_kwargs and setting profiles_dir. Have you tried to make the changes @Noah Holm suggested and test DbtShellTask again?
m
gahh I just followed all of this and am still getting same error 
fatal: Not a dbt project (or any of the parent directories). Missing dbt_project.yml file
I checked the image locally thought and it did have that dbt_project.yml file
so you use dbt_kwargs to set all of the params and also profiles_dir? Don't they each do the same thing?
j
Sounds like DbtShellTask is not executing in the directory that contains your dbt_project.yml. Have you made sure the helper_script is setting the right directory where dbt_project.yml exists? Maybe you can set the parameter return_all=True to see if you get any extra information.
m
yeah I tested it locally and all of the files are located in the dockerfile
when I ssh in and do ls all the files in directory are there
however when I do dbt run in the image I get
Copy code
Encountered an error while reading the project:
  ERROR: Runtime Error
  Could not find profile named 'winc_dev'
j
maybe you can try and run ls at the end of the helper_script to see what you get
m
But those seems to be too different errors
j
ok that is a different error, since it is not being able to find your profile
The way I went was to create the profile using DbtShellTask, but maybe it does not apply to your case.
m
that's what I am currently doing but the location that that is executing in doesn't seem to match up with the WORKDIR I defined in my Dockerfile
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by