https://prefect.io logo
Title
m

Mitchell Bregman

06/14/2022, 1:58 AM
Follow up to the above thread - we are running into an issue now with connecting dbt to our Redshift warehouse in Cloud... have verified that all the environment variables are working correctly, do we need to install anything explicitly in our Docker image?
01:13:51  Running with dbt=1.1.0
01:13:51  Partial parse save file not found. Starting full parse.
01:13:52  Found 9 models, 75 tests, 0 snapshots, 0 analyses, 196 macros, 0 operations, 0 seed files, 1 source, 0 exposures, 0 metrics
01:13:52
01:13:52  Encountered an error:
Database Error
  could not connect to server: No such file or directory
  	Is the server running locally and accepting
  	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5439"?
1
why is it looking for some local postgres?
a

Anna Geller

06/14/2022, 11:22 AM
I believe you may get more help with respect to how to run dbt in Docker when you ask in dbt Slack, but let me try to help • yes, you would need to have dbt-redshift and probably also dbt-postgres installed • can you try running your flow with dbt first only in a virtual environment and confirm that it works? • then, can you try to copy all the files you used locally in a virtual environment within a Docker image, build it, run this container locally, exec into it, inspect the file structure from the container and do dbt run from within to see if this is not Docker network issue? also maybe Matthias and Leon can help you more here
m

Mitchell Bregman

06/14/2022, 1:35 PM
to answer your points: • I am installing
dbt-redshift
and
dbt-postgres
• my flow works in a venv • i think i need to include some additional steps in my Docker image, but i think this is the culprit. do i need to install
postgresql
on top of the drivers? here is what my Dockerfile looks like
FROM prefecthq/prefect:0.15.12-python3.8 as builder

RUN apt-get update && apt-get install --no-install-recommends --no-upgrade -y \
    libpq-dev \
    git \
    ssh \
    curl -y && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip3 install --upgrade pip
RUN pip3 install --no-cache-dir -r requirements.txt

FROM prefecthq/prefect:0.15.12-python3.8

RUN apt-get update && apt-get install --no-install-recommends --no-upgrade -y \
    libpq-dev && \
    rm -rf /var/lib/apt/lists/*


COPY --from=builder /usr/local/lib/python3.8/site-packages/ /usr/local/lib/python3.8/site-packages/
# For DBT pip install, we need psycopg2 not the binary.
# See: <https://github.com/dbt-labs/dbt-core/blob/a2e040f389e64cc78a8e2d262c3ab2802b56044b/plugins/postgres/setup.py#L22>
RUN DBT_PSYCOPG2_NAME=psycopg2 pip3 install dbt-redshift --no-cache-dir --no-binary dbt-postgres

ENV PYTHONPATH="$PYTHONPATH:src/"

WORKDIR /app

COPY src src
COPY creds.yaml .
COPY job_template.yaml .
COPY VERSION .
m

Matthias

06/14/2022, 6:20 PM
My advise would be similar to what Anna suggested: try to run the flow locally, but run it from a shell within the container. There you can inspect if the structure of the file system looks ok etc. From what I can see in the logs, it might be an issue with your
profiles.yaml
🙌 1
If you need inspiration, here is a working example https://github.com/MatthiasRoels/prefect-examples/tree/main/dbt-integration
❤️ 1