a

    Alex Furrier

    1 year ago
    I'm trying to understand how to create a Flow of Flows (dependent flows) from flows I've been running locally. Currently I can run the flows in succession using a Makefile where each file invoked by python contains a Flow and a flow.run() wrapped with CLI param inputs:
    test-flow-of-flows:
    	@echo 'Running flow A'
    	@python flows/flow_a.py \
    		param1=foo \
    	    param2=bar 
    	@python flows/flow_b.py \
    		param1=foo \
    	    param2=bar 		
    	@python flows/flow_c.py \
    		param1=baz \
    	    param2=bar
    I would like to combine those into a single Flow run with shared parameters passed to the flow runs. That seems to be what's described in this documentation. That mentions registering flows using the orchestration API to specific projects. So far I've been running flows without that. Is there any way to create a flow of flows with local importing of flows or do they have to be registered with the orchestration API to do so?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Alex Furrier, normally when we talk about Flow of Flows, they are done with the orchestration API. There is a Prefect
    StartFlowRun
    task that the main flow will use to call Flows that are already registered. I think the way you have right now would be the only way, or if you had a python script that did the same thing by calling the shell commands. You’re aware that Prefect Cloud gives 10000 task runs for free each month right?
    a

    Alex Furrier

    1 year ago
    Yeah my team is planning on using the orchestration API but it's a work in progress at the moment. Registering the flows would take some work refactoring things so I wasn't sure if there was a temporary workaround I could implement in the mean time
    Kevin Kho

    Kevin Kho

    1 year ago
    I see, Yeah it might just be a matter of having a “main” Python flow and you can use the
    ShellTask
    Prefect has to start thesr scripts