I find myself frequently wanting to chain some tasks together and parametrize the set of tasks. And then have other tasks depend on the set of tasks. I am currently using the imperative api to add the set of tasks to the flow, and then function that adds it to the flow returns the first and last task, and then I set dependencies on those first and last tasks. It is not a super clean solution, but it works. Dagster - another python workflow manager has a concept of

composite_tasks

which let's you treat a handful of tasks as one task in your workflow. Something similar in

prefect

I think could be super useful. Just wanted to kindly share my thoughts. And I am curious if this is something

prefect

is thinking about or if there are any upcoming plans that address this problem?

Hi @Ryan Connolly - you can chain a list of tasks together using

Flow.chain

(https://docs.prefect.io/api/latest/core/flow.html#prefect-core-flow-flow-chain), and then set dependencies on the first and last as you are currently doing — does that simplify your workflow pattern?

I guess I am looking for a way to compose reusable workflows into higher level prefect workflows. really a task that can take a set of arguments and is composed of multiple tasks and that task can be set as dependencies within the larger workflow the

Flow.chain

doesn't seem great for managing

switch

or

ifelse

statements (even though those are just tasks themselves and could be chained) or

keword_tasks

if you need data to flow between tasks. I am able to make it work quite well with the initial methodology I described... but I was just thinking it would be nice to have a first class primitive for this something like a

@composite_task

decorator that now allows me to set dependencies on the single composite task and make it more reusable and confined Again, it is not a barrier or blocker to accomplishing any given set of tasks, it is just something I find myself wanting

Would you expect Prefect to track the tasks within the “composite” task or are you wanting them to be considered a single Task from Prefect’s perspective?

I would want the

composite_task

to just be a container for the composed task, but I guess I would still expect to be able to configure all the retries and state handlers at the individual task level

Gotcha gotcha, that makes sense - thanks! This might be a very easy thing to implement so I’ll look into it and get back to you

Thank you @Chris White and @Jeremiah I really appreciate you all taking the time to hear me out

@Chris White This feature would also be very useful for us, was wondering if you had any updates on whether it is on the horizon