https://prefect.io
Join Slack
Hello, I was wondering what the correct pattern is to run multiple flows that are dependent on the s...
# prefect-server
w
Hello, I was wondering what the correct pattern is to run multiple flows that are dependent on the same tasks via different parameters. I know that you can parameterize your flows but I need to create new runs for a list of passed in parameters in a more automatic way then registering the parameters in the flow themselves.
k
Hey @William Clark, not sure I understand the question but you can do:
Copy code
with Flow(...) as flow:
    x = Parameter("x", True)
    with case(x, True):
        create_flow_run(...)
and you can parameterize the start of sub flow runs this way. Is this what you mean?
w
This is bare example but this is what it looks like
Copy code
bucket_1 = 'bucket'

model_1 = 'cool_model_1'

model_2 = 'cool_model_2'

@task('Get Model')
def get_model_s3(bucket_name, model_name) -> Path:
    "Gets a trained model from S3 and stores it in a temporary directory"
    return Path(f'/tmp/{model_name}')

@task('Evaluate Model')
def evaluate_model(model):
    "Evaluates a model"
    return model.evaluate()



with Flow('Evalulate Model'):
    bucket_name = Parameter('bucket_name')
    model_name = Parameter('model_name')
    
    model = get_model_s3(bucket_name, model_name)
    evaluate_model(model)


flow.run()
The tasks are always going to be the same, get_model_s3 and evaluate_model. I need to pass the model names which could be stored in a json file. So for every model name found in the json file register and create a new flow run.
I think create_flow_run is what I'm looking for
Essentially have a parent flow that will create new flow run of the above flow for each model name found in a json file.
k
I think it would be something like this:
Copy code
@task
def make_param_dict(model_name):
    return {"model_name": model_name}

with Flow(...) as flow:
    models = Parameter("models", ["1","2","3"])
    param_dict = make_param_dict.map(models)
    create_flow_run.map(..., parameters=param_dict)
w
I see does .map only accept one parameter per task?
k
Map can take in a list of arguments and then there will be 1 task run for each set of items. Have you seen the docs about this? I can link it if you haven’t yet
w
Got it, and yes, I sure have! I am still a little lost on some of how some of the pieces work together but I'll figure it out shortly! Thanks for answering all my questions and being such a champion for this community!
k
Ah ok. 
.map()
can have multiple parameters per task to answer your question. And of course! No problem!
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by