Hi friends, I have a bunch of ETLs with a similiar set of preliminary tasks and closing tasks, and so I wrote a

get_etl_flow

function that takes a callback function that sets up the ETL-specific tasks, and thus defines and returns a flow by setting up these common tasks and calling the callback function. Then I have a

register_flow

function and a

run_flow

function that calls

get_etl_flow

and registers or runs the flow returned, respectively. Each group of related ETLs are grouped into a prefect project in a single Python script, and

register_flow

is called for each ETL in the main function. If I want to run a particular flow manually, I have to change that particular flow's

register_flow

call to

run_flow

and comment out the other

register_flow

calls, which is cumbersome. How can I make use of the new CLI for running flows such that my infrastructure can still register flows and run them via an agent, while I can for debugging etc run individual flows using agentless execution, without having to comment out code, despite there being multiple flows defined in a single file?

For our project we created our one command line that can do all of that using the prefect client. This allow as well to properly parameterize or setup dask cluster. It maybe overkilk however it streamline our development experience a lot

Hey @Amanda Wee, first with agentless execution, you would still need to register the flow and then run it on local. If you just do

prefect run -p file_name.py

, then you would be prompted to select the flow you want to run so if you script is something like:

import prefect
from prefect import Flow, task

@task
def abc(x):
    return x

@task
def bcd(x):
    return x

with Flow("1") as flow:
    abc(1)

with Flow("2") as flow2:
    abc(2)

You can do:

prefect run -p file_name.py --name "2"