Hi! I have a loop which runs task_a and then task_b, but want to skip task_b depending on the result of task_a. See code below:
Copy code
for i in range(x):
task_a_result = task_a.submit(i)
if task_a_result.result() == "do_task_b":
task_b.submit(task_a_result)
It seems that doing task_a.result() blocks the for loop, is there a way to do this logic concurrently? Or I should have task_b handle different types of results of task_a instead?
d
Dominic Tarro
12/11/2023, 4:09 PM
Something worth trying.
If you use async and a result queue, you can have task_a add the result to the queue. Another async func (not a task) is regularly polling for items in the queue. When it picks up a result that meets the condition, it submits task_b with the result.
Dominic Tarro
12/11/2023, 4:11 PM
Basically just need to create a queue in the flow, submit that queue as an argument to task a, then await a func that does the polling. You need to implement some logic that tells the task_b submitter to stop polling and continue on with the flow.
Dominic Tarro
12/11/2023, 4:13 PM
Frankly, it may be simpler to create an async subflow that does this instead. Just need to write some orchestration code to asynchronously start and await all of the subflows.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.