https://prefect.io logo
Title
y

Yang Ruan

05/16/2022, 5:19 PM
Hi, another question about steps. We'd like one task to unzip folders in gcs, and another task to process each file in the folder. Can that be done where the second task fans out (multiprocess)?
k

Kevin Kho

05/16/2022, 5:43 PM
Yes. For Prefect 1, see mapping and for Prefect 2, you can just use a for loop and call new tasks. You just need to call
.result()
on the first task to make it a Python object before you loop
y

Yang Ruan

05/16/2022, 6:05 PM
using a for loop will do the tasks sequentially right?
k

Kevin Kho

05/16/2022, 6:06 PM
Not quite. The default TaskRunner is ConcurrentTaskRunner so if they are independent, they will run concurrrently
y

Yang Ruan

05/16/2022, 6:11 PM
oh cool!