John Ramirez
03/11/2020, 7:40 PMswitch
and ifelse
Zachary Hughes
03/11/2020, 7:50 PMJohn Ramirez
03/11/2020, 7:51 PMLaura Lorenz (she/her)
2 days ago
Is it possible to implement your situation such that the dataframe is returned from the upstream task already split up into two, like {'a': df1, 'b': df2}, and then the downstream tasks index that result like task1(result['a'] and task2(result['b'] ? Basically doing the splitting itself in the upstream task
Laura Lorenz (she/her)
2 days ago
the switch task operates on a dictionary, but also it is for only doing one task tree of a set of mutually exclusive task trees, but it sounds like you want to run the task tree for group A AND group B, not just one or the other
daily_pull = df[df['adjustment_factor'] == 1.0]
full_refresh = df[df['adjustment_factor'] == 2.0]
return {
'daily': daily_pull[COLUMN_MAPPING['quote_cols']],
'full_refresh': full_refresh[COLUMN_MAPPING['quote_cols']]
}
Zachary Hughes
03/11/2020, 8:00 PM@task
def generate_dataframe():
return {"portion_a": 123, "portion_b": 456}
@task
def process_dataframe_portion_a():
# do something
return ...
@task
def process_dataframe_portion_b():
# do something
return ...
with Flow("switch-flow") as flow:
dataframe = generate_dataframe()
portion_a_results = process_dataframe_portion_a(dataframe["portion_a"])
portion_b_results = process_dataframe_portion_b(dataframe["portion_b"])
John Ramirez
03/11/2020, 8:07 PMZachary Hughes
03/11/2020, 8:38 PM