https://prefect.io logo
#prefect-community
Title
# prefect-community
j

Jason

05/12/2022, 9:14 PM
I have a task that outputs a tuple of data_frames, and other tasks that output individual data_frames. I'm trying to think of a good way to reduce them into one tuple of data_frames so I can use a single map task over them. So far, I've tried extending a list and coercing back to a tuple, and a few other methods that have failed. There's likely a parsimonious way to do this that I'm missing
a

Anna Geller

05/12/2022, 9:16 PM
How would you do it without Prefect? Can you share a code snippet showing that?
j

Jason

05/12/2022, 9:16 PM
I wonder if I can just unpack the tuple to combine it:
Copy code
task.map([*tupled_data_frames, data_frame2, data_frame3])
a

Anna Geller

05/12/2022, 9:19 PM
unpacking won't work because in 1.0 you can only construct a DAG in your Flow - you can try but I'm 95% sure it won't work
j

Jason

05/12/2022, 9:20 PM
Flow run SUCCESS! All data frames were written to Snowflake. That's neat, I didn't think that would work.
a

Anna Geller

05/12/2022, 9:22 PM
oh wow, could you share your Flow code? just the flow constructor without actual tasks
j

Jason

05/12/2022, 9:23 PM
Sure:
Copy code
with Flow(
    FLOW_NAME,
    storage=Config.storage(project=PROJECT, file_name=FILE_NAME),
    result=S3Result(bucket="EDITED"),
    run_config=Config.run(),
) as flow:

    with case(save_snowflake, True):  # type: ignore

        load_snowflake.map(
            data_payload=[*reservations, owners, properties, amenities],
            drop=unmapped(drop_snowflake),
        )
I am passing an explicit
nout
in the
reservations
task, which is perhaps why it works? idk.
a

Anna Geller

05/12/2022, 9:25 PM
thanks for sharing!
3 Views