https://prefect.io logo
Title
w

wonsun

11/02/2022, 8:36 AM
Hi all, I can't find where the error occurred. I'm using prefect 1.0. As I looked for examples to set the dependency between flows, I saw an article written by anna geller. (Link) Some time has passed since she wrote this article, so I changed
FlowRunTask
to
StartFlowRun
(reference link) among the methods written in that article to create the same situation. Below attached scripts, it is the case that two flows(flow_waveforms and flow_pipeline) that are executed well have already been created, and parent_flow is created to sequentially execute those flows.
# parent_flow.py
import prefect
from prefect.tasks.prefect.flow_run import StartFlowRun

with Flow('mother_flow') as flow:
    first_job = StartFlowRun(flow_name='flow_waveforms',
                             project_name='InterFlow_Dependencies',
                             wait=True)
    second_job = StartFlowRun(flow_name='flow_pipeline',
                              project_name='InterFlow_Dependencies',
                              wait=True)
    first_job.set_downstream(second_job)

if __name__ == '__main__':
    flow.register("InterFlow_Dependencies")
# flow_waveforms.py
import prefect
from prefect import Flow, task

@task
def task1():
    something
    return one, two
@task
def task2():
    something

with Flow('flow_waveforms') as flow:
    var1, var2 = task1()
    finish = task2()

flow.run_config = UniversalRun(env={'PREFECT__CLOUD__HEARTBEAT_MODE':'thread'})
flow.register('InterFlow_Dependencies')
# flow_pipeline.py
import prefect
from prefect import Flow, task

@task
def task3():
    something

with Flow('flow_pipelines') as flow:
    task3()

flow.register('InterFlow_Dependencies')
If i excuted mother_flow.py in this case, prefect responds following error.(image) Any idea where I went wrong? Or could you tell me if there is an easy way to define dependencies between flows other than this way? Thx.
a

Anna Geller

11/02/2022, 11:50 AM
This can be solved much easier in Prefect 2 using subflows. I wonder, do you have a chance to implement it directly in v2?
if you want to go with this approach in v1, I recommend taking it more step by step: first register and run each child flow from the UI and ensure they are working, then trigger only one child flow from the parent flow and so on
w

wonsun

11/03/2022, 12:44 AM
Yeah, prefect 2 seems to be easier to implement in this job. However, since the implementation of the pipeline that was initially conceived using version 1 is almost completion.So, I think it would be better to wrap up version 1 and migration to version 2 for now and I'm still not perfect at using prefect. 😥 Each "flow" works well through several tests and i checked that in UI already. This may be the first condition you mentioned. However, I didn't understand what you said "trigger only one child flow from the parent flow". As in the above situation, if two flows are registered in one parent flow, is it correct to say that only the first flow is executed in the parent flow?
a

Anna Geller

11/03/2022, 12:54 AM
I'm still not perfect at using prefect
me neither 🙌 registration shouldn't really be in the flow, it's a separate step - this might be helpful to understand what I mean https://github.com/anna-geller/orchestrator-pattern/tree/master/simple_parent_child_example
:marvin: 1
w

wonsun

11/03/2022, 1:13 AM
Oh.. I'll check that link. Thank you so much for all the help, whenever I was in trouble like this! Especially, many of your articles are very helpful for people who are new to prefect, and also thanks to other prefect's engineer in Slack community. 💙
a

Anna Geller

11/03/2022, 11:58 AM
Of course, we've got you! 🙌