https://prefect.io logo
Channels
Powered by
n

Neeraj Sharma

11/05/2020, 5:54 AM
Hello, I want to create dask cluster on Azure. Is their any API's available in Prefect which can create Dask Cluster in Azure .
n

nicholas

11/05/2020, 2:40 PM
Hi @Neeraj Sharma - we don't have any Azure-specific tasks to create a Dask Cluster in Azure but that sounds like a great feature request/contribution! For reference, you can check out the ever-growing Task Library, and more specifically the Azure tasks.
j

james.lamb

11/05/2020, 3:45 PM
I think this could be done without adding a new task, and probably should. The 
dask/dask-cloudprovider
project has a hook for creating a cluster on Azure ML: https://cloudprovider.dask.org/en/latest/azure.html You could adopt this example to use that 😀 https://docs.prefect.io/orchestration/execution/dask_cloud_provider_environment.html#overview
upvote 1
n

Neeraj Sharma

11/06/2020, 11:38 AM
Yes i tried AzureMLCluster API but again there is one more issue. We have datastore in datalake gen2. When I passed datalake gen2 datastore in AzureMLCluster API, got below error.'
Copy code
"error": {
        "code": "UserError",
        "message": "Type <class 'azureml.data.azure_data_lake_datastore.AzureDataLakeGen2Datastore'> is not supported for inputs."
    }
So basically I am going backward :)
j

james.lamb

11/06/2020, 3:30 PM
ha ah ok, I see! It looks like there is an open pull request in 
dask-cloudprovider
right now that is removing the idea of datastores completely from that class: https://github.com/dask/dask-cloudprovider/pull/144/files If you have the time and a small reproducible example of how you're calling 
dask_cloudprovider.AzureMLCluster
, I think you'd have good luck opening an issue there: https://github.com/dask/dask-cloudprovider/issues It seems like there's been a lot of Azure activity in that project the last few days. Another option you have is to try using 
dask-kubernetes
(https://kubernetes.dask.org/en/latest/) on AKS (Azure Kubernetes Service).
n

Neeraj Sharma

11/09/2020, 6:22 AM
Thank you James
Will raise it with explanation of how I am using
Powered by