Kotra Pali
04/30/2020, 12:53 PMRename(source='cola', target='colb').
Transpose(source=[...]),
Map(colname="a", values={3:4, 5:7}),
Assign(condition='colb>2', target="cola", value=100)
...
We have about 100k of these per single processing and there are some dependencies between commands (e.g. you can see that cola
depends on values of colb
in the last Assign
command and we are able to parse these dependencies in advance).
What I am thinking about is to create `prefect.Task`s from these transformation funcs and create a graph. That means a DAG with like 100k nodes with some edges at least (but each of them very lightweight). Do you think it's a good fit/idea for prefect? Can it handle such huge DAGs, or is it rather design for smaller ones?Laura Lorenz (she/her)
04/30/2020, 1:06 PMKotra Pali
04/30/2020, 1:12 PM