I have what is probably a very simple question. How do I write a Pandas DataFrame to Snowflake when...

Brian Roepke

10/26/2023, 3:35 PM

I have what is probably a very simple question. How do I write a Pandas DataFrame to Snowflake when utilizing the snowflake connector block? This code

Copy code

from prefect import task, flow, get_run_logger
from prefect_snowflake.database import SnowflakeConnector
from prefect_snowflake.database import snowflake_query
from snowflake.connector.pandas_tools import pd_writer

snowflake_connector_block = SnowflakeConnector.load("my_snowflake")

data_df = #some other code that created a DF

data_df.to_sql(table_name, con=snowflake_connector_block, if_exists="append", method=pd_writer)

Throw the Error - which is related to using SQL Alchemy engine - which I don't want to necessarily because of credentials

Copy code

AttributeError: 'SnowflakeConnector' object has no attribute 'cursor'

Brian Roepke

10/26/2023, 4:14 PM

I also tried this exact example: https://github.com/PrefectHQ/prefect-snowflake/tree/main#access-underlying-snowflake-connection and gives the same error

Brian Roepke

10/26/2023, 4:33 PM

Got it!

Brian Roepke

10/26/2023, 4:33 PM

there was an error in the docs

Brian Roepke

10/26/2023, 4:34 PM

Copy code

def snowflake_write_pandas_flow():
    connector = SnowflakeConnector.load("my-block")
    with connector.get_connection() as connection:
        table_name = "TABLE_NAME"
        ddl = "NAME STRING, NUMBER INT"
        statement = f'CREATE TABLE IF NOT EXISTS {table_name} ({ddl})'
        with connector.cursor() as cursor:
            cursor.execute(statement)

on the second to last line - connector needs to be "connection"

Brian Roepke

10/26/2023, 4:58 PM

PR for docs approved: https://github.com/PrefectHQ/prefect-snowflake/pull/89

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.