Is this a good place to ask questions about the prefect-snowflake task? It uses

.fetchall()

which isn't usable for large datasets. Any other attempts we've done to use similar logic but using a cursor fails. What is it about the prefect execution environment that prevents us from using snowflake cursors?

There’s a refactor coming, but in the meantime, you can get the cursor manually as you suggested; however I don’t think it’s pickleable so you if you use some Prefect features, you cannot pass it from task to task. Also, you can also use the LIMIT statement in your query

If not pickleable, does that mean all the connection work needs to be in a single task? Or that we must package our flows using the script-based mechanism rather than the pickling mechanism? I ask because we tried putting everything into a single task, but we were getting python errors like

unknown opcode

and hanging snowflake connections

Yes I think within a single task if you use caching, if I’m not mistaken; if not I think it should work.

I had hoped that would be the case, but isn't the behavior we are seeing

Can you share a reproducible example?

I can work on it