https://prefect.io logo
#prefect-community
Title
# prefect-community
r

Ryan Connolly

01/27/2020, 3:59 PM
I find myself frequently wanting to chain some tasks together and parametrize the set of tasks. And then have other tasks depend on the set of tasks. I am currently using the imperative api to add the set of tasks to the flow, and then function that adds it to the flow returns the first and last task, and then I set dependencies on those first and last tasks. It is not a super clean solution, but it works. Dagster - another python workflow manager has a concept of
composite_tasks
which let's you treat a handful of tasks as one task in your workflow. Something similar in
prefect
I think could be super useful. Just wanted to kindly share my thoughts. And I am curious if this is something
prefect
is thinking about or if there are any upcoming plans that address this problem?
c

Chris White

01/27/2020, 4:13 PM
Hi @Ryan Connolly - you can chain a list of tasks together using
Flow.chain
(https://docs.prefect.io/api/latest/core/flow.html#prefect-core-flow-flow-chain), and then set dependencies on the first and last as you are currently doing — does that simplify your workflow pattern?
r

Ryan Connolly

01/27/2020, 4:38 PM
I guess I am looking for a way to compose reusable workflows into higher level prefect workflows. really a task that can take a set of arguments and is composed of multiple tasks and that task can be set as dependencies within the larger workflow the
Flow.chain
doesn't seem great for managing
switch
or
ifelse
statements (even though those are just tasks themselves and could be chained) or
keword_tasks
if you need data to flow between tasks. I am able to make it work quite well with the initial methodology I described... but I was just thinking it would be nice to have a first class primitive for this something like a
@composite_task
decorator that now allows me to set dependencies on the single composite task and make it more reusable and confined Again, it is not a barrier or blocker to accomplishing any given set of tasks, it is just something I find myself wanting
c

Chris White

01/27/2020, 4:44 PM
Would you expect Prefect to track the tasks within the “composite” task or are you wanting them to be considered a single Task from Prefect’s perspective?
r

Ryan Connolly

01/27/2020, 6:20 PM
I would want the
composite_task
to just be a container for the composed task, but I guess I would still expect to be able to configure all the retries and state handlers at the individual task level
c

Chris White

01/27/2020, 6:22 PM
Gotcha gotcha, that makes sense - thanks! This might be a very easy thing to implement so I’ll look into it and get back to you
r

Ryan Connolly

01/27/2020, 6:24 PM
Thank you @Chris White and @Jeremiah I really appreciate you all taking the time to hear me out
👍 2
b

Ben Fogelson

03/25/2020, 8:31 PM
@Chris White This feature would also be very useful for us, was wondering if you had any updates on whether it is on the horizon