https://prefect.io logo
Powered by
Title
m

Madison Schott

08/12/2021, 10:02 PM
Has anyone else ever followed this tutorial when using DbtShellTask? I've created my Dockerfile to be similar but am having issues with my dbt files https://www.lejimmy.com/distributed-data-pipelines-with-aws-ecs-fargate-and-prefect-cloud/
n

Noah Holm

08/13/2021, 12:55 PM
I’m running dbt with Prefect on ECS Fargate but not using Docker storage. Is dbt installed in your image that you run the flow in? What’s the issue with the dbt files? Are they included in your image as well or are you downloading them from somewhere externally when executing the flow?
👍 1
m

Madison Schott

08/13/2021, 1:32 PM
Yeah dbt is installed but it’s saying the directory that I’m copying is not a dbt project even though I have my dockerfile inside the project
I’m trying to copy them from the directory I have the dockerfile stored in- not sure if that is right though- my flow which created the docket image is in another file
n

Noah Holm

08/13/2021, 1:41 PM
Can you show some examples? It sounds like something is just off with how you define things in the Dockerfile, or use your built image
m

Madison Schott

08/13/2021, 1:54 PM
# specify a base image
FROM prefecthq/prefect:latest

WORKDIR dbt_snowflake/

COPY . .

# install all dependencies
RUN pip install -U pip
RUN pip install dbt==0.17.2
that's my Dockerfile and this is what my folder structure looks like
STORAGE = Docker(registry_url='xx',
                 image_name='prefect-flows',
                 dockerfile='/Users/madisonschott/dbt_snowflake/DockerFile')
that's in the file for my flow when I register it
but then also my DbtShellTask looks like this so I'm not sure how to handle that when trying to use the Docker image 
user_profile_dbt_run = dbt_task(helper_script = "cd /Users/madisonschott/dbt_snowflake",
                             command='dbt run')
n

Noah Holm

08/13/2021, 2:09 PM
If you build your image in the root of 
dbt_snowflake
the path that you cd into shouldn’t have your local mac’s path. I think you should 
"cd dbt_snowflake"
You could check locally where your dbt project is within the image with 
docker run --rm -it <imagename> sh
, that way you should see where you need to cd
I also had to juggle with the profiles a bit. This is my code 
DBT_REPO_PATH = "/tmp/dbt"

dbt = DbtShellTask(
    helper_script=f"cd {DBT_REPO_PATH}",
    log_stderr=True,
    log_stdout=True,
    profiles_dir=DBT_REPO_PATH,
    stream_output=True,
    checkpoint=False,
)
Then I run it in the flow with 
dbt(command="dbt run")
m

Madison Schott

08/13/2021, 2:20 PM
If I'm using 
dbt_kwargs
do I even need to specify the profiles_dir?
Does the Dockerfile look right to you? For how my folder structure is?
n

Noah Holm

08/13/2021, 2:22 PM
Not sure, can’t remember the turns I took when testing myself. Try it without first
Yes I think the Dockerfile is OK. Your fix will probably be on the flow side of things. I’m not sure of the details when the image is built by Prefect and not yourself. That’s why I’d suggest looking inside the docker image to see where your dbt project is located in that file tree
j

J. Martins

08/13/2021, 2:28 PM
If it helps, I am also using dbt_kwargs and setting profiles_dir. Have you tried to make the changes @Noah Holm suggested and test DbtShellTask again?
m

Madison Schott

08/13/2021, 2:49 PM
gahh I just followed all of this and am still getting same error 
fatal: Not a dbt project (or any of the parent directories). Missing dbt_project.yml file
I checked the image locally thought and it did have that dbt_project.yml file
so you use dbt_kwargs to set all of the params and also profiles_dir? Don't they each do the same thing?
j

J. Martins

08/13/2021, 3:06 PM
Sounds like DbtShellTask is not executing in the directory that contains your dbt_project.yml. Have you made sure the helper_script is setting the right directory where dbt_project.yml exists? Maybe you can set the parameter return_all=True to see if you get any extra information.
m

Madison Schott

08/13/2021, 3:07 PM
yeah I tested it locally and all of the files are located in the dockerfile
when I ssh in and do ls all the files in directory are there
however when I do dbt run in the image I get 
Encountered an error while reading the project:
  ERROR: Runtime Error
  Could not find profile named 'winc_dev'
j

J. Martins

08/13/2021, 3:09 PM
maybe you can try and run ls at the end of the helper_script to see what you get
m

Madison Schott

08/13/2021, 3:09 PM
But those seems to be too different errors
j

J. Martins

08/13/2021, 3:10 PM
ok that is a different error, since it is not being able to find your profile
The way I went was to create the profile using DbtShellTask, but maybe it does not apply to your case.
m

Madison Schott

08/13/2021, 3:16 PM
that's what I am currently doing but the location that that is executing in doesn't seem to match up with the WORKDIR I defined in my Dockerfile
#prefect-serverJoin Slack
Powered by