If I want to map over a task that takes several arguments. e.g.
def some_task(id:str, old: Dict, new: Dict)
and I have data as
[{"id":"some_id", "old":{...}, "new":{...}}, ...]
is there some easy way to explode the dict into the tasks, or is it best to write some wrapper task whose only responsibility is to take the data and pass it to the intended task.
m
Mariia Kerimova
09/09/2021, 1:33 PM
Hello Filip! It feels like you are looking for something like mapping, right?
k
Kevin Kho
09/09/2021, 1:56 PM
Hey @Filip Lindvall, you can’t unpack Tuple and List inside a Flow block if that is what you are asking. You need another task because Tuple and List unpacking execute immediately during build time.
If the
Tuple
,
Dict
, or
List
is already defined (not a task output), then you can unpack it. If they are the output of a Task, then you need an intermediate task to unpack or you need to pass
{"id":"some_id", "old":{...}, "new":{...}}
in the task and unpack it in there.
f
Filip Lindvall
09/10/2021, 12:45 PM
Okay, yeah. I built some small things to support it, but requires some custom tasks to get it working, thanks anyway 🙂
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.