https://prefect.io logo
Title
a

Adam Roderick

07/11/2019, 8:15 PM
Design question. I'm in a multi-tenant situation, so I could use a single flow and maps. Or I could use a parameterized flow and run it once per tenant. What are some of the tradeoffs of either approach?
c

Chris White

07/11/2019, 9:03 PM
Excellent question! - the consequences of task failure within a single tenant are different in the two approaches; for example, your entire flow will presumably be marked failed if a single tenant fails, and you’ll have to do some introspection to determine which tenant failed - with mapping, note that all results from the mapped tasks will need to fit in memory
I’m sure there are some other tradeoffs that I’m not thinking of, but those come to my mind first
a

Adam Roderick

07/11/2019, 9:15 PM
Got it, thanks! So I'm leaning towards parameterized flow that I call once per tenant, because of the clean history and troubleshooting. Let's say I want each tenant to have a flow run once per day. Do I create a single "master" or wrapper flow that queries to get an up-to-date tenant list, then runs a sub-run for each tenant (forgive my terminology)?
c

Chris White

07/12/2019, 5:52 AM
Sorry for the delayed response; that’s definitely an option - you’ll probably run into the question of how to kick them off asynchronously / in parallel. I’ve seen people submit each individual flow run in a separate process that is kicked off from a master flow, so that’s certainly one option --> however, this very very quickly runs into all of the can of worms of “orchestration”: - what happens if one of the processes dies? - how do you know when they’re finished? etc.
a

Adam Roderick

07/12/2019, 2:20 PM
In the tutorials I've walked through, calling flow.run() with the next schedule run of let's say tomorrow causes the process to just pause until then--meaning only 1 workflow at a time. It seems like kicking off many workflows with schedules is not possible without some external process managing it..... prefect cloud? 🙂
☝️ 1
c

Chris White

08/05/2019, 9:39 PM
@Marvin archive “Single Flow with mapping vs multiple parametrized Flows”