Leonard Marcq

09/16/2020, 7:50 PM
Hi there, I managed to deploy a flow that spawns a bunch of mapped tasks that run on a Dask cluster on Fargate with
, it works pretty well and was much easier to set up than I expected. But it seems that the cluster that gets spawned runs the Dask tasks into the default VPC in AWS (and its associated security group), but I would prefer those tasks to run in another VPC than the default one. I did not find much info on the network config for that in the doc or in Dask's Cloud Provider documentation - ideally I would like to pass specific security groups / subnets and all that for the Dask cluster. (1) Is there a way of passing a network config to the Fargate Dask cluster through the
interface (or some other way)? (2) Does it matter what kind of agent picks up that flow? From what I understand, the only thing the agent does in this case is spawn a cluster with the Scheduler and the Worker nodes in it and check that the tasks completed at the end (and maybe tear down the Dask cluster?), so I guess any agent could do the job or does it specifically have to also be a Fargate agent for some reason?


09/16/2020, 7:56 PM
Hey @Leonard Marcq for the Dask Cloud Provider config I would probably recommend using a LocalEnvironment and passing in a dask_cloudprovider
as the executor (example of how to pass in the executor here) The agent you use does not necessarily matter because you are shipping out your tasks’ work to a dask cluster meaning that you could do something like use the LocalAgent and have your tasks still execute on a remote dask cluster running in Fargate since the execution is based on the executor.

Leonard Marcq

09/16/2020, 8:05 PM
@josh - Cool. Thank you, I will see if I can get that to work. Thanks for the explanation about the agent as well