Hi everyone — I have a task that is mapped over a list, and I have a subsequent task that must only run after all the branches of the previous task map are done, however the tasks are not data dependant. I’m trying to specify “upstream_tasks” but it’s giving me a weird error as it appears it is trying to run the task?
k
Kevin Kho
05/26/2021, 1:51 PM
Hey again, I believe this thread is exactly your issue and the solution is to just add a reduce step at the end
Kevin Kho
05/26/2021, 1:53 PM
As long as you have the reduce at the end, this should work. The reduce can even just be a logging step. that returns None.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.