https://prefect.io logo
Title
j

John Ramirez

03/11/2020, 7:40 PM
Hey - does anyone have a code sample for Control Flow task such as
switch
and
ifelse
z

Zachary Hughes

03/11/2020, 7:50 PM
j

John Ramirez

03/11/2020, 7:51 PM
No I was having this conversation in a different thread
Laura Lorenz (she/her)
  2 days ago

Is it possible to implement your situation such that the dataframe is returned from the upstream task already split up into two, like {'a': df1, 'b': df2}, and then the downstream tasks index that result like task1(result['a'] and task2(result['b'] ? Basically doing the splitting itself in the upstream task
Laura Lorenz (she/her)
  2 days ago

the switch task operates on a dictionary, but also it is for only doing one task tree of a set of mutually exclusive task trees, but it sounds like you want to run the task tree for group A AND group B, not just one or the other
I understand what Laura is saying but not familiar enough with prefect to implement it
I have split results into two separate pandas data frames and want to send the results to different tasks
daily_pull = df[df['adjustment_factor'] == 1.0]
    full_refresh = df[df['adjustment_factor'] == 2.0]
    return {
        'daily': daily_pull[COLUMN_MAPPING['quote_cols']],
        'full_refresh': full_refresh[COLUMN_MAPPING['quote_cols']]
    }
z

Zachary Hughes

03/11/2020, 8:00 PM
Okay, gotcha. What I think she's proposing is something along the lines of:
@task
def generate_dataframe():
    return {"portion_a": 123, "portion_b": 456}

@task
def process_dataframe_portion_a():
    # do something
    return ...

@task
def process_dataframe_portion_b():
    # do something
    return ... 

with Flow("switch-flow") as flow:
    dataframe = generate_dataframe()
    portion_a_results = process_dataframe_portion_a(dataframe["portion_a"])
    portion_b_results = process_dataframe_portion_b(dataframe["portion_b"])
j

John Ramirez

03/11/2020, 8:07 PM
ok that is what i though but I was getting errors cause I was passing a mapped object
thank you for your help
👍 1
z

Zachary Hughes

03/11/2020, 8:38 PM
Sure thing!