https://prefect.io
Join Slack
Is there a way to "package" up a set of tasks into a super task that can be re-used easily? For exam...
# ask-community
m
Is there a way to "package" up a set of tasks into a super task that can be re-used easily? For example, we have a few tasks that usually get used in sequence: validate some pandas dataframe, grab some secrets from AWS, use those secrets/credentials to insert the dataframe into a database
✅ 1
o
Could you use a subflow?
m
ya, i guess i could. but it seems like more overhead
id also prefer to see everything in the main flow in the UI -- not a subflow run
o
Yeah, I'd prefer that too. Maybe there will be additional UI features related to subflows in the future, that'd probably make them a good fit for this kind of use case. I can't think of any other ways of solving your problem, since you can't call tasks from tasks.
j
Subflows are probably the literal solution to your question, as they create a single “virtual task” in the parent flow for all their component tasks but still let you “drill down” to the component tasks if you want. Avoiding subflows, two thoughts occur to me: 1. you could distribute your tasks as part of a larger function (just a regular function, not a flow) that you call inside your flows. This would then pass the inputs to your tasks in sequence. This aligns with your goal of seeing all of your tasks in the UI, but only calling one function. 2. if your goal is to package them into a single task (still without a subflow), you should be able to recover their underlying functions as 
your_task.fn(…)
. This aligns with your goal of packaging them as a super task (which could have its own retry settings, for example), but would not let you see them individually in the UI.
m
Interesting @Jeremiah . What would 1 look like
j
Something like this --
Copy code
from prefect import flow, task

@task
def task_1(x):
    return x + 1.5

@task
def task_2(y):
    return y + 10.5

def super_task(x):
    """
    This function could be imported from anywhere and 
    called inside any flow; it's a normal function
    """
    y = task_1(x)
    z = task_2(y)
    return z

@flow
def my_flow():
    return super_task(17.0)
super_task
being the function you’d give people
m
ok cool. can i do that in v1? I was under the impression that everything in the flow must be a task
j
Yes, I expect this would work in V1 — every computation in the flow must be a task, but here we’re using 
super_task
purely as a convenience function; it’s equivalent to calling 
task_1
and 
task_2
yourself
(though my code is for v2 and e.g. subflows are only in v2)
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by