https://prefect.io
Join Slack
Hello, I want to create dask cluster on Azure. Is their any API's available in Prefect which can cre...
# ask-community
n
Hello, I want to create dask cluster on Azure. Is their any API's available in Prefect which can create Dask Cluster in Azure .
n
Hi @Neeraj Sharma - we don't have any Azure-specific tasks to create a Dask Cluster in Azure but that sounds like a great feature request/contribution! For reference, you can check out the ever-growing Task Library, and more specifically the Azure tasks.
j
I think this could be done without adding a new task, and probably should. The 
dask/dask-cloudprovider
project has a hook for creating a cluster on Azure ML: https://cloudprovider.dask.org/en/latest/azure.html You could adopt this example to use that 😀 https://docs.prefect.io/orchestration/execution/dask_cloud_provider_environment.html#overview
upvote 1
n
Yes i tried AzureMLCluster API but again there is one more issue. We have datastore in datalake gen2. When I passed datalake gen2 datastore in AzureMLCluster API, got below error.'
Copy code
"error": {
        "code": "UserError",
        "message": "Type <class 'azureml.data.azure_data_lake_datastore.AzureDataLakeGen2Datastore'> is not supported for inputs."
    }
So basically I am going backward :)
j
ha ah ok, I see! It looks like there is an open pull request in 
dask-cloudprovider
right now that is removing the idea of datastores completely from that class: https://github.com/dask/dask-cloudprovider/pull/144/files If you have the time and a small reproducible example of how you're calling 
dask_cloudprovider.AzureMLCluster
, I think you'd have good luck opening an issue there: https://github.com/dask/dask-cloudprovider/issues It seems like there's been a lot of Azure activity in that project the last few days. Another option you have is to try using 
dask-kubernetes
(https://kubernetes.dask.org/en/latest/) on AKS (Azure Kubernetes Service).
n
Thank you James
Will raise it with explanation of how I am using
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by