Will Milner

05/22/2020, 3:20 PM
quick question on running flows in a distributed system like kubernetes. When I run a flow, do all tasks in that flow run in the same execution environment, or could some tasks run on one environment and the rest in a different envrionment?


05/22/2020, 3:25 PM
As it currently stands all of your tasks will execute using the same execution environment. If you do something in your tasks like each one creates its own k8s job that performs the work then you technically could send them out to their own environments If you are using a Dask executor with your environment then all of the tasks will be executed on that Dask cluster you specified

Will Milner

05/22/2020, 3:48 PM
thanks for the clarification