https://prefect.io logo
a

Ashton

01/23/2022, 8:29 PM
hey y'all, do tasks support returning generators? Currently, when I return a generator from a task, the results don't get returned or get picked up in the next task. Couldn't find anything in the docs about it
e

emre

01/23/2022, 9:19 PM
Hey, could you post a specific example? My basic test works well:
Copy code
from prefect import Flow, task, unmapped


@task(log_stdout=True)
def printl(elem, r):
    print(f"{elem} is produced by {r}")


@task
def get_gen(end):
    return range(end)


with Flow("Test 1") as flow:
    g = get_gen(15)
    printl.map(g, unmapped(g))

if __name__ == "__main__":
    flow.run()
My bad, turns out range works but a custom generator fails the receiving task.
a

Anna Geller

01/23/2022, 9:47 PM
@Ashton can you explain a bit more what type of problem do you try to solve when you return a generator in your task? There might be an alternative approach depending on what you try to do - do you feed it into a map operation?
a

Ashton

01/23/2022, 10:23 PM
Thanks for the help! I'm just yielding data from sequential API calls with different parameters. I solved this problem by appending the data into a list and returning it. I was just testing using a generator and noticed that the generator doesn't do anything in a prefect task. Not a huge issue, but was just curious.
a

Anna Geller

01/23/2022, 10:36 PM
That makes a lot of sense, thanks for sharing your use case!
k

Kevin Kho

01/24/2022, 3:34 AM
Yes you are right that tasks can’t output generators
3 Views