https://prefect.io logo
a

An Hoang

02/02/2022, 5:38 PM
I have a
flow_A
with
task1, task2... task10
. Now I want to construct
flow_B
with
task1, task2.... task 20
and
task11_result = task11(task10_result)
. How do I add the tasks of
flow_A
as upstream tasks for
flow_B
. I know I can do
create_flow_run
, but I do not want monitoring for
task1...task10
in
flow_B
Kind of like:
Copy code
with Flow("flow_A") as flow:
    flow_A_param1 = Parameter("flow_A_param1")
    ...
    flow_A_param_n = Parameter(...)
    ...
    task1 = ....
    ...
    task10_result = task10(...)

with Flow("flow_B") as flow:
    flow_A_param1 = Parameter(...)
    ...
    flow_A_param_n = Parameter(...)
    
    flow_A_result = flow_A.run(parameters = {"flow_A_param1": flow_A_param1,...., "flow_A_param_n": flow_A_param_n}) #dont want monitoring for these tasks
    task10_result = flow_A_result.result[flow_A.get_task(task10)]
    task11_result = task11(task10_result)
    ...
    task20_result = task20(...)
where `flow_B`'s parameters are just `flow_A`'s parameters. But in the example above I always have to repeat the parameters of
flow_A
in
flow_B
and keep them both up to date with each other. Is there a better way to do this?
k

Kevin Kho

02/02/2022, 6:07 PM
Oh man this is hard. This is all Prefect Core right? So there is no GraphQL API to query?
a

An Hoang

02/02/2022, 6:10 PM
Yes
flow_A
would be ran using prefect core and
flow_B
would be ran using Prefect cloud for monitoring
I guess the first thing to tackle is how to make flow share the same parameters and then add tasks that build upon those parameters
k

Kevin Kho

02/02/2022, 6:13 PM
You would need a function to create the Flows and return them at the very least to make it dynamic based on inputs
3 Views