Hello, I was wondering what the correct pattern is...
# prefect-server
w
Hello, I was wondering what the correct pattern is to run multiple flows that are dependent on the same tasks via different parameters. I know that you can parameterize your flows but I need to create new runs for a list of passed in parameters in a more automatic way then registering the parameters in the flow themselves.
k
Hey @William Clark, not sure I understand the question but you can do:
Copy code
with Flow(...) as flow:
    x = Parameter("x", True)
    with case(x, True):
        create_flow_run(...)
and you can parameterize the start of sub flow runs this way. Is this what you mean?
w
This is bare example but this is what it looks like
Copy code
bucket_1 = 'bucket'

model_1 = 'cool_model_1'

model_2 = 'cool_model_2'

@task('Get Model')
def get_model_s3(bucket_name, model_name) -> Path:
    "Gets a trained model from S3 and stores it in a temporary directory"
    return Path(f'/tmp/{model_name}')

@task('Evaluate Model')
def evaluate_model(model):
    "Evaluates a model"
    return model.evaluate()



with Flow('Evalulate Model'):
    bucket_name = Parameter('bucket_name')
    model_name = Parameter('model_name')
    
    model = get_model_s3(bucket_name, model_name)
    evaluate_model(model)


flow.run()
The tasks are always going to be the same, get_model_s3 and evaluate_model. I need to pass the model names which could be stored in a json file. So for every model name found in the json file register and create a new flow run.
I think create_flow_run is what I'm looking for
Essentially have a parent flow that will create new flow run of the above flow for each model name found in a json file.
k
I think it would be something like this:
Copy code
@task
def make_param_dict(model_name):
    return {"model_name": model_name}

with Flow(...) as flow:
    models = Parameter("models", ["1","2","3"])
    param_dict = make_param_dict.map(models)
    create_flow_run.map(..., parameters=param_dict)
w
I see does .map only accept one parameter per task?
k
Map can take in a list of arguments and then there will be 1 task run for each set of items. Have you seen the docs about this? I can link it if you haven’t yet
w
Got it, and yes, I sure have! I am still a little lost on some of how some of the pieces work together but I'll figure it out shortly! Thanks for answering all my questions and being such a champion for this community!
k
Ah ok.
.map()
can have multiple parameters per task to answer your question. And of course! No problem!