Channels
#prefect-server
Title
p
Paulo Benatto
06/17/2021, 8:59 AMHi guys, how r u doing? I'm new to prefect and i was reading: https://docs.prefect.io/core/idioms/parallel.html and i think im missing something 😏. I would like to build a list of tasks to be executed in parallel. See this dummy example:
Copy code
@task
def create_task_list(ids):
l = []
for i in ids:
l.append(ReferralUID())
return l
@task
def generate_ids():
return ["id1", "id2", "id3"]
with Flow("parallel-execution") as flow:
ids = generate_ids()
list_of_tasks = create_task_list(ids)
request_referral = RequestReferralDetail()
# list_of_tasks should be a list, but i'm returning a task. how to inject a list on bind?
request_referral.bind(request_details=list_of_tasks, flow=flow)
flow.visualize()
flow.run(executor=LocalDaskExecutor())r
Rob Fowler
06/17/2021, 9:57 AM Copy code
request_referral.map(list_of_items, unampped(static_argument)would run multiple instances of the RequestReferralDetail for each item in list_of_items
although from your language you want want to run multiple different tasks in parallel which happens automagically.
My example is running a task multiple times in parallel operating on a list of items to be operated in parallel
p
Paulo Benatto
06/17/2021, 10:05 AMlovely @Rob Fowler i will read the page now 🙇
r
Rob Fowler
06/17/2021, 10:16 AMjust a ps. If you are wanting more threads you can increase them:
flow.executor = LocalDaskExecutor(workers=30, ..
I find processes more stable for large numbers of workers but YMMV
🙌 1
p
Paulo Benatto
06/17/2021, 12:44 PMit does work, however i have to call
runRequestReferralDetail Copy code
class RequestReferralDetail(Task):
def run(self, request_details):
if isinstance(request_details, Task):
request_details.run()2 Views