We have some flows that could run locally for small datasets. What is the recommended way to invoke a flow and dynamically change the associated TaskRunner?
✅ 1
👍 1
t
Taylor Curran
09/15/2022, 6:08 PM
Hmm. Hi Tony, that is a good question.
Can you give me more information on your use case? When you say dynamically change the task runner, are you more interested in Dask vs Ray? Or are you more interested in certain tasks being run asynchronously and others being run synchronously?
t
Tony Piazza
09/15/2022, 6:12 PM
I'm thinking ConcurrentTaskRunner vs DaskTaskRunner.
Or dask.distributed.LocalCluster vs coiled.Cluster
Depending on the characteristics of the dataset, we might choose one type of cluster vs another.
t
Taylor Curran
09/15/2022, 6:18 PM
The first thing that comes to my mind is building 2 flows with different task runners and using get_client() create_flow_run_from_deployment() to dynamically call one flow over the other… though this is not great in terms of DRY code..
I am going to phone a friend who knows more about task runners. Please give us some time.
s
Serina
09/15/2022, 6:25 PM
You could also do some logic in the flow like “if dataset is large, run this subflow w/ specified taskrunner, else run this other subflow w/ other taskrunner.” if both subflows would be exactly the same except of the TaskRunner, you could also consider making the name of the TaskRunner a block and just passing it in