OK, So I am trying to use the DaskRunner with a for loop to concurrently run a task over a list.
Copy code
@flow(name="Subscriptions Flow",
task_runner=DaskTaskRunner())
def main():
*snip*
pages_list = get_pages_list(client, "subscriptions", params)
for page in pages_list:
item_list = get_items_list(client, "subscriptions", page)
if __name__ == "__main__":
flow_result = main()
print(flow_result)
But I get a RuntimeError from the multiprocessing. Can I get some help on what I am doing wrong with the DaskRunner. I want to parallel run over what is in
pages_list
k
Kevin Kho
06/09/2022, 2:53 PM
Can you share the error here?
t
Tim Enders
06/09/2022, 2:54 PM
I hadn't saved a typo fix facepalm
Tim Enders
06/09/2022, 2:54 PM
Now I get maximum recursion depth exceeded... which is a whole other can of worms
🤦♂️ 1
z
Zanie
06/09/2022, 2:56 PM
You’ll need to iterate over
pages_list.result()
Zanie
06/09/2022, 2:57 PM
Not sure why you’re getting a recursion error though
t
Tim Enders
06/09/2022, 2:57 PM
Oh that is my API client failing to get a token... which is not a Prefect problem
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.