https://prefect.io logo
Title
t

Tim Enders

06/09/2022, 2:51 PM
OK, So I am trying to use the DaskRunner with a for loop to concurrently run a task over a list.
@flow(name="Subscriptions Flow",
      task_runner=DaskTaskRunner())
def main():
   *snip*

    pages_list = get_pages_list(client, "subscriptions", params)
    for page in pages_list:
        item_list = get_items_list(client, "subscriptions", page)
 

if __name__ == "__main__":
    flow_result = main()
    print(flow_result)
But I get a RuntimeError from the multiprocessing. Can I get some help on what I am doing wrong with the DaskRunner. I want to parallel run over what is in
pages_list
k

Kevin Kho

06/09/2022, 2:53 PM
Can you share the error here?
t

Tim Enders

06/09/2022, 2:54 PM
I hadn't saved a typo fix 🤦
Now I get maximum recursion depth exceeded... which is a whole other can of worms
🤦‍♂️ 1
z

Zanie

06/09/2022, 2:56 PM
You’ll need to iterate over
pages_list.result()
Not sure why you’re getting a recursion error though
t

Tim Enders

06/09/2022, 2:57 PM
Oh that is my API client failing to get a token... which is not a Prefect problem