https://prefect.io logo
Title
m

Marc Lipoff

08/24/2022, 8:26 PM
Is there a way to "package" up a set of tasks into a super task that can be re-used easily? For example, we have a few tasks that usually get used in sequence: validate some pandas dataframe, grab some secrets from AWS, use those secrets/credentials to insert the dataframe into a database
1
o

Oscar Björhn

08/24/2022, 8:29 PM
Could you use a subflow?
m

Marc Lipoff

08/24/2022, 8:31 PM
ya, i guess i could. but it seems like more overhead
id also prefer to see everything in the main flow in the UI -- not a subflow run
o

Oscar Björhn

08/24/2022, 8:37 PM
Yeah, I'd prefer that too. Maybe there will be additional UI features related to subflows in the future, that'd probably make them a good fit for this kind of use case. I can't think of any other ways of solving your problem, since you can't call tasks from tasks.
j

Jeremiah

08/24/2022, 9:45 PM
Subflows are probably the literal solution to your question, as they create a single “virtual task” in the parent flow for all their component tasks but still let you “drill down” to the component tasks if you want. Avoiding subflows, two thoughts occur to me: 1. you could distribute your tasks as part of a larger function (just a regular function, not a flow) that you call inside your flows. This would then pass the inputs to your tasks in sequence. This aligns with your goal of seeing all of your tasks in the UI, but only calling one function. 2. if your goal is to package them into a single task (still without a subflow), you should be able to recover their underlying functions as
your_task.fn(…)
. This aligns with your goal of packaging them as a super task (which could have its own retry settings, for example), but would not let you see them individually in the UI.
m

Marc Lipoff

08/24/2022, 9:50 PM
Interesting @Jeremiah . What would 1 look like
j

Jeremiah

08/24/2022, 9:58 PM
Something like this --
from prefect import flow, task

@task
def task_1(x):
    return x + 1.5

@task
def task_2(y):
    return y + 10.5

def super_task(x):
    """
    This function could be imported from anywhere and 
    called inside any flow; it's a normal function
    """
    y = task_1(x)
    z = task_2(y)
    return z

@flow
def my_flow():
    return super_task(17.0)
super_task
being the function you’d give people
m

Marc Lipoff

08/25/2022, 2:08 PM
ok cool. can i do that in v1? I was under the impression that everything in the flow must be a task
j

Jeremiah

08/25/2022, 7:54 PM
Yes, I expect this would work in V1 — every computation in the flow must be a task, but here we’re using
super_task
purely as a convenience function; it’s equivalent to calling
task_1
and
task_2
yourself
(though my code is for v2 and e.g. subflows are only in v2)