Ryan Connolly
01/27/2020, 3:59 PMcomposite_tasks
which let's you treat a handful of tasks as one task in your workflow.
Something similar in prefect
I think could be super useful.
Just wanted to kindly share my thoughts. And I am curious if this is something prefect
is thinking about or if there are any upcoming plans that address this problem?Chris White
01/27/2020, 4:13 PMFlow.chain
(https://docs.prefect.io/api/latest/core/flow.html#prefect-core-flow-flow-chain), and then set dependencies on the first and last as you are currently doing — does that simplify your workflow pattern?Ryan Connolly
01/27/2020, 4:38 PMFlow.chain
doesn't seem great for managing switch
or ifelse
statements (even though those are just tasks themselves and could be chained) or keword_tasks
if you need data to flow between tasks.
I am able to make it work quite well with the initial methodology I described... but I was just thinking it would be nice to have a first class primitive for this
something like a @composite_task
decorator that now allows me to set dependencies on the single composite task and make it more reusable and confined
Again, it is not a barrier or blocker to accomplishing any given set of tasks, it is just something I find myself wantingChris White
01/27/2020, 4:44 PMRyan Connolly
01/27/2020, 6:20 PMcomposite_task
to just be a container for the composed task, but I guess I would still expect to be able to configure all the retries and state handlers at the individual task levelChris White
01/27/2020, 6:22 PMRyan Connolly
01/27/2020, 6:24 PMBen Fogelson
03/25/2020, 8:31 PM