Tomer Cagan03/02/2022, 12:25 PM
in Prefect 1.0 or
for Prefect 2.0, then you wouldn't rely on Prefect-specific parallelism but rather the parallelism provided by your tool depending on how you call it in your tasks and flows. Having said that, Prefect is open-source so nothing stops you from building your own custom executors. If you would want to take a stab at it, you would essentially need a class that satisfies the Executor interface (in 1.0) including those functions: • submit • map • wait similarly to Python's
Tomer Cagan03/02/2022, 1:43 PM
- to run things sequentially in the same process, •
- to run things in parallel using async, •
- to offload the task run execution to a Dask cluster - this probably answers your question best, since this Dask cluster could be executed e.g. on a Kubernetes cluster essentially separating task run execution across a cluster of resources/nodes Executors in Prefect 1.0 are analogous.