kevin
03/11/2022, 4:08 AMselect * from table
to snowflake using a SnowflakeQuery task. It appears that this task uses fetchall()
to return this data into memory: https://github.com/PrefectHQ/prefect/blob/5d2732a30563591410cb11fe0f7e7dfe65cc5669/src/prefect/tasks/snowflake/snowflake.py#L186
I expect that this causes performance issues with extremely large queries so I am wondering what Prefect Cloud’s tolerance for this. Ideally I think it would be preferable to lazy load query results and/or allow for query pagination? Perhaps there’s an architecture limitation I’m overlooking? I’d appreciate any insight 🙂Kevin Kho
03/11/2022, 4:12 AMkevin
03/11/2022, 4:15 AMKevin Kho
03/11/2022, 4:17 AM# Execute a statement that will generate a result set.
sql = "select * from t"
cur.execute(sql)
# Fetch the result set from the cursor and deliver it as the Pandas DataFrame.
for df in cur.fetch_pandas_batches():
my_dataframe_processing_function(df)
kevin
03/11/2022, 4:21 AMKevin Kho
03/11/2022, 4:21 AMfetch_pandas_all()
kevin
03/11/2022, 4:23 AMKevin Kho
03/11/2022, 4:24 AM