https://prefect.io logo
#prefect-community
Title
# prefect-community
a

Adam Roderick

12/20/2022, 10:41 PM
Is this a good place to ask questions about the prefect-snowflake task? It uses
.fetchall()
which isn't usable for large datasets. Any other attempts we've done to use similar logic but using a cursor fails. What is it about the prefect execution environment that prevents us from using snowflake cursors?
a

Andrew Huang

12/20/2022, 11:18 PM
There’s a refactor coming, but in the meantime, you can get the cursor manually as you suggested; however I don’t think it’s pickleable so you if you use some Prefect features, you cannot pass it from task to task. Also, you can also use the LIMIT statement in your query
a

Adam Roderick

12/20/2022, 11:20 PM
If not pickleable, does that mean all the connection work needs to be in a single task? Or that we must package our flows using the script-based mechanism rather than the pickling mechanism? I ask because we tried putting everything into a single task, but we were getting python errors like
unknown opcode
and hanging snowflake connections
a

Andrew Huang

12/20/2022, 11:21 PM
Yes I think within a single task if you use caching, if I’m not mistaken; if not I think it should work.
a

Adam Roderick

12/20/2022, 11:22 PM
I had hoped that would be the case, but isn't the behavior we are seeing
a

Andrew Huang

12/20/2022, 11:22 PM
Can you share a reproducible example?
a

Adam Roderick

12/20/2022, 11:22 PM
I can work on it
👀 1