Hi, any help would be greatly appreciated. I've been tasked with assisting upgrading Prefect 1.x code to Prefect 2, and for the most part the migration has been smooth. I have a flow that has a task that raises a LOOP signal, and I'm struggling to figure out how to properly handle/recreate this functionality in prefect 2. Is there a good way to do this same thing in Prefect 2?
Copy code
raise LOOP(message=f"Status Run {n}={v_run_code}", result=dict(n=n + 1, code=v_run_code))
✅ 1
z
Zanie
04/18/2023, 3:52 PM
Hey! If you need to loop in Prefect 2 you can just write a loop using standard Python instead of raising a signal.
Zanie
04/18/2023, 3:54 PM
e.g.
Copy code
for n in range(n):
my_task(n)
or
Copy code
n = 0
while my_task(n) is None:
n += 1
Zanie
04/18/2023, 3:54 PM
in the second, you’d return
None
to indicate another iteration
j
Jason Damiani
04/18/2023, 3:57 PM
@Zanie in Prefect 1 I have a flow that uses LOOP signal within a task that is run as a mapped task. What would be the best option in Prefect 2? A nested loop? (I'm essentially using LOOP to dynamically spawn sequential tasks for the given mapped task)
z
Zanie
04/18/2023, 5:28 PM
Hm that’s a bit trickier. A loop should work though like
Copy code
futures = my_task.map(range(100))
for i, future in enumerate(futures):
if future.result() is None:
my_task.submit(future, i + 1)
Zanie
04/18/2023, 5:29 PM
You’d need another loop that ensures everything is done too if you want multiple loops to happen
Zanie
04/18/2023, 5:30 PM
Can you share some more about your use-case? I haven’t seen someone loop in a mapped task like this before. cc @justabill / @Chris White
🙏 1
j
Jason Damiani
04/18/2023, 5:49 PM
The use case is essentially API pagination. We map over a set of IDs, each mapped task run then paginates over api calls until a stop condition is achieved. Using LOOP, each individual API call can be retried using the standard prefect machinery
👍 2
z
Zanie
04/18/2023, 7:08 PM
I think in the current world that would be best implemented with concurrent subflows (for each id) then a loop executing the tasks
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.