We have some flows that could run locally for small datasets. What is the recommended way to invoke a flow and dynamically change the associated TaskRunner?

Hmm. Hi Tony, that is a good question. Can you give me more information on your use case? When you say dynamically change the task runner, are you more interested in Dask vs Ray? Or are you more interested in certain tasks being run asynchronously and others being run synchronously?

I'm thinking ConcurrentTaskRunner vs DaskTaskRunner.

The first thing that comes to my mind is building 2 flows with different task runners and using get_client() create_flow_run_from_deployment() to dynamically call one flow over the other… though this is not great in terms of DRY code.. I am going to phone a friend who knows more about task runners. Please give us some time.

You could also do some logic in the flow like “if dataset is large, run this subflow w/ specified taskrunner, else run this other subflow w/ other taskrunner.” if both subflows would be exactly the same except of the TaskRunner, you could also consider making the name of the TaskRunner a block and just passing it in

Here is an example that might help you get started with subflows having different task runners… warning that the example is slightly out of date: https://discourse.prefect.io/t/can-my-subflow-use-a-different-task-runner-than-my-parent-flow/101

@Taylor Curran this looks promising. I will study it tonight. Thanks!