m

    Mitchell Bregman

    3 months ago
    Follow up to the above thread - we are running into an issue now with connecting dbt to our Redshift warehouse in Cloud... have verified that all the environment variables are working correctly, do we need to install anything explicitly in our Docker image?
    01:13:51  Running with dbt=1.1.0
    01:13:51  Partial parse save file not found. Starting full parse.
    01:13:52  Found 9 models, 75 tests, 0 snapshots, 0 analyses, 196 macros, 0 operations, 0 seed files, 1 source, 0 exposures, 0 metrics
    01:13:52
    01:13:52  Encountered an error:
    Database Error
      could not connect to server: No such file or directory
      	Is the server running locally and accepting
      	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5439"?
    why is it looking for some local postgres?
    Anna Geller

    Anna Geller

    3 months ago
    I believe you may get more help with respect to how to run dbt in Docker when you ask in dbt Slack, but let me try to help • yes, you would need to have dbt-redshift and probably also dbt-postgres installed • can you try running your flow with dbt first only in a virtual environment and confirm that it works? • then, can you try to copy all the files you used locally in a virtual environment within a Docker image, build it, run this container locally, exec into it, inspect the file structure from the container and do dbt run from within to see if this is not Docker network issue? also maybe Matthias and Leon can help you more here
    m

    Mitchell Bregman

    3 months ago
    to answer your points: • I am installing
    dbt-redshift
    and
    dbt-postgres
    • my flow works in a venv • i think i need to include some additional steps in my Docker image, but i think this is the culprit. do i need to install
    postgresql
    on top of the drivers? here is what my Dockerfile looks like
    FROM prefecthq/prefect:0.15.12-python3.8 as builder
    
    RUN apt-get update && apt-get install --no-install-recommends --no-upgrade -y \
        libpq-dev \
        git \
        ssh \
        curl -y && \
        rm -rf /var/lib/apt/lists/*
    
    COPY requirements.txt .
    RUN pip3 install --upgrade pip
    RUN pip3 install --no-cache-dir -r requirements.txt
    
    FROM prefecthq/prefect:0.15.12-python3.8
    
    RUN apt-get update && apt-get install --no-install-recommends --no-upgrade -y \
        libpq-dev && \
        rm -rf /var/lib/apt/lists/*
    
    
    COPY --from=builder /usr/local/lib/python3.8/site-packages/ /usr/local/lib/python3.8/site-packages/
    # For DBT pip install, we need psycopg2 not the binary.
    # See: <https://github.com/dbt-labs/dbt-core/blob/a2e040f389e64cc78a8e2d262c3ab2802b56044b/plugins/postgres/setup.py#L22>
    RUN DBT_PSYCOPG2_NAME=psycopg2 pip3 install dbt-redshift --no-cache-dir --no-binary dbt-postgres
    
    ENV PYTHONPATH="$PYTHONPATH:src/"
    
    WORKDIR /app
    
    COPY src src
    COPY creds.yaml .
    COPY job_template.yaml .
    COPY VERSION .
    Matthias

    Matthias

    3 months ago
    My advise would be similar to what Anna suggested: try to run the flow locally, but run it from a shell within the container. There you can inspect if the structure of the file system looks ok etc. From what I can see in the logs, it might be an issue with your
    profiles.yaml