https://prefect.io logo
Title
t

Tony Piazza

09/15/2022, 5:51 PM
We have some flows that could run locally for small datasets. What is the recommended way to invoke a flow and dynamically change the associated TaskRunner?
1
👍 1
t

Taylor Curran

09/15/2022, 6:08 PM
Hmm. Hi Tony, that is a good question. Can you give me more information on your use case? When you say dynamically change the task runner, are you more interested in Dask vs Ray? Or are you more interested in certain tasks being run asynchronously and others being run synchronously?
t

Tony Piazza

09/15/2022, 6:12 PM
I'm thinking ConcurrentTaskRunner vs DaskTaskRunner.
Or dask.distributed.LocalCluster vs coiled.Cluster
Depending on the characteristics of the dataset, we might choose one type of cluster vs another.
t

Taylor Curran

09/15/2022, 6:18 PM
The first thing that comes to my mind is building 2 flows with different task runners and using get_client() create_flow_run_from_deployment() to dynamically call one flow over the other… though this is not great in terms of DRY code.. I am going to phone a friend who knows more about task runners. Please give us some time.
s

Serina

09/15/2022, 6:25 PM
You could also do some logic in the flow like “if dataset is large, run this subflow w/ specified taskrunner, else run this other subflow w/ other taskrunner.” if both subflows would be exactly the same except of the TaskRunner, you could also consider making the name of the TaskRunner a block and just passing it in
😛anda-dancing: 1
t

Taylor Curran

09/15/2022, 6:33 PM
Here is an example that might help you get started with subflows having different task runners… warning that the example is slightly out of date: https://discourse.prefect.io/t/can-my-subflow-use-a-different-task-runner-than-my-parent-flow/101
Also, @Tony Piazza, have you checked out our Fugue collection? It was built for exactly your use case!
t

Tony Piazza

09/15/2022, 11:11 PM
@Taylor Curran this looks promising. I will study it tonight. Thanks!
:marvin: 1