https://prefect.io logo
Join Slack
Powered by
Hi guys, how r u doing? I'm new to prefect and i w...
# prefect-server
p
Hi guys, how r u doing? I'm new to prefect and i was reading: https://docs.prefect.io/core/idioms/parallel.html and i think im missing something 😏. I would like to build a list of tasks to be executed in parallel. See this dummy example:
Copy code
@task
def create_task_list(ids):
    l = []
    for i in ids:
        l.append(ReferralUID())
    return l

@task
def generate_ids():
    return ["id1", "id2", "id3"]



with Flow("parallel-execution") as flow:

    ids = generate_ids()
    
    list_of_tasks = create_task_list(ids)

    request_referral = RequestReferralDetail()

    # list_of_tasks should be a list, but i'm returning a task. how to inject a list on bind?
    request_referral.bind(request_details=list_of_tasks, flow=flow)

    flow.visualize()
    flow.run(executor=LocalDaskExecutor())
Thanks
r
Copy code
request_referral.map(list_of_items, unampped(static_argument)
would run multiple instances of the RequestReferralDetail for each item in list_of_items
although from your language you want want to run multiple different tasks in parallel which happens automagically. My example is running a task multiple times in parallel operating on a list of items to be operated in parallel
p
lovely @Rob Fowler i will read the page now 🙇
r
just a ps. If you are wanting more threads you can increase them: flow.executor = LocalDaskExecutor(workers=30, .. I find processes more stable for large numbers of workers but YMMV
🙌 1
p
it does work, however i have to call 
run
method inside of 
RequestReferralDetail
Copy code
class RequestReferralDetail(Task):
    def run(self, request_details):
        if isinstance(request_details, Task):
            request_details.run()
I think I still missing something, probably i will read more doc 😄 Thanks ur help mate
2 Views
Open in Slack
PreviousNext
Powered by