Hi there, I managed to deploy a flow that spawns a bunch of mapped tasks that run on a Dask cluster on Fargate with

DaskCloudProviderEnvironment

, it works pretty well and was much easier to set up than I expected. But it seems that the cluster that gets spawned runs the Dask tasks into the default VPC in AWS (and its associated security group), but I would prefer those tasks to run in another VPC than the default one. I did not find much info on the network config for that in the doc or in Dask's Cloud Provider documentation - ideally I would like to pass specific security groups / subnets and all that for the Dask cluster. (1) Is there a way of passing a network config to the Fargate Dask cluster through the

DaskCloudProviderEnvironment

interface (or some other way)? (2) Does it matter what kind of agent picks up that flow? From what I understand, the only thing the agent does in this case is spawn a cluster with the Scheduler and the Worker nodes in it and check that the tasks completed at the end (and maybe tear down the Dask cluster?), so I guess any agent could do the job or does it specifically have to also be a Fargate agent for some reason?

Hey @Leonard Marcq for the Dask Cloud Provider config I would probably recommend using a LocalEnvironment and passing in a dask_cloudprovider

FargateCluster

as the executor (example of how to pass in the executor here) The agent you use does not necessarily matter because you are shipping out your tasks’ work to a dask cluster meaning that you could do something like use the LocalAgent and have your tasks still execute on a remote dask cluster running in Fargate since the execution is based on the executor.

@josh - Cool. Thank you, I will see if I can get that to work. Thanks for the explanation about the agent as well