Question about working with large(ish) amounts of snowflake data with prefect_snowflake. Wondering why we cant seem to use the

fetch_many

(or anything really) from the

SnowflakeConnector

with a yield instead of return.

@task
def fetch_data(block_name: str) -> list:
    all_rows = []

    with SnowflakeConnector.load(block_name) as connector:
        while True:
            batch = connector.fetch_many("SELECT * FROM MY_BIG_TABLE", size=10000)

            if len(batch) == 0:
                break

            all_rows.append(batch) # why append????

    return all_rows

This works fine but if I try doing something like this:

@task
def fetch_data(block_name: str):
    with SnowflakeConnector.load(block_name) as connector:
        while True:
            batch = connector.fetch_many("SELECT * FROM MY_BIG_TABLE", size=10000)

            if len(batch) == 0:
                break

            yield batch

I get a ton of errors steaming from the SnowflakeConnector as

'coroutine' object does not support the context manager protocol

We are able to

yield

from prefect tasks in general and wondering why these seems to really not like being a generator

Tasks don't like yields. You could use a yield to return values inside a task, but not to return values to the flow. I don't know the technical parlance, but tasks in Prefect are designed to return a single definite result. Generators don't work with the task model.