Neeraj Sharma
11/05/2020, 5:54 AMnicholas
james.lamb
11/05/2020, 3:45 PMdask/dask-cloudprovider
project has a hook for creating a cluster on Azure ML: https://cloudprovider.dask.org/en/latest/azure.html
You could adopt this example to use that 😀
https://docs.prefect.io/orchestration/execution/dask_cloud_provider_environment.html#overviewNeeraj Sharma
11/06/2020, 11:38 AMNeeraj Sharma
11/06/2020, 11:39 AM"error": {
"code": "UserError",
"message": "Type <class 'azureml.data.azure_data_lake_datastore.AzureDataLakeGen2Datastore'> is not supported for inputs."
}
Neeraj Sharma
11/06/2020, 11:40 AMjames.lamb
11/06/2020, 3:30 PMdask-cloudprovider
right now that is removing the idea of datastores completely from that class: https://github.com/dask/dask-cloudprovider/pull/144/files
If you have the time and a small reproducible example of how you're calling dask_cloudprovider.AzureMLCluster
, I think you'd have good luck opening an issue there: https://github.com/dask/dask-cloudprovider/issues
It seems like there's been a lot of Azure activity in that project the last few days.
Another option you have is to try using dask-kubernetes
(https://kubernetes.dask.org/en/latest/) on AKS (Azure Kubernetes Service).Neeraj Sharma
11/09/2020, 6:22 AMNeeraj Sharma
11/09/2020, 6:23 AM