Tomer Cagan
03/02/2022, 12:25 PMAnna Geller
03/02/2022, 12:38 PMLocalExecutor
in Prefect 1.0 or SequentialTaskRunner
for Prefect 2.0, then you wouldn't rely on Prefect-specific parallelism but rather the parallelism provided by your tool depending on how you call it in your tasks and flows.
Having said that, Prefect is open-source so nothing stops you from building your own custom executors. If you would want to take a stab at it, you would essentially need a class that satisfies the Executor interface (in 1.0) including those functions:
• submit
• map
• wait
similarly to Python's concurrent.futures
executor's interface.Tomer Cagan
03/02/2022, 1:43 PMAnna Geller
03/02/2022, 1:55 PMSequentialTaskRunner
- to run things sequentially in the same process,
• ConcurrentTaskRunner
- to run things in parallel using async,
• DaskTaskRunner
- to offload the task run execution to a Dask cluster - this probably answers your question best, since this Dask cluster could be executed e.g. on a Kubernetes cluster essentially separating task run execution across a cluster of resources/nodes
Executors in Prefect 1.0 are analogous.Kevin Kho
03/02/2022, 2:18 PM