Our team is working on a PoC in Prefect 2. We deployed Prefect Orion and Agent using the helm chart from Github. We tried to hook up the existing Dask cluster in k8s with Prefect 2. But I can't find any doc about the Dask deployment setup other than specifying DaskTaskExecutor in the flow script. Should the Dask cluster hooking up done in the Agent?

Hi @Ricky Zhang, the agent needs to be able to communicate with the Dask cluster so as long as where ever the agent lives can communicate with the dask cluster it should be fine.

I think DaskTaskExecutor is Prefect 1. For Prefect 2, there’s prefect-dask here https://prefecthq.github.io/prefect-dask/

@Kalise Richmond, I work with Ricky. Can you point us to the best doc explaining how to get the agent to communicate with a Dask cluster (in k8s)?

@John Lewis That looks correct. To run the tasks on the dask cluster, you would specify the DaskTaskRunner. https://docs.prefect.io/concepts/task-runners/#running-tasks-on-dask Andrew's link to the prefect-dask integration has a lot of examples as well. Is this not how you were expecting to use Dask with Prefect? If not, can you provide more details on what you are trying to do or what's not working?

@Kalise Richmond, thanks for the reply. When using dask-kubernetes, you can set KUBECONFIG and it will connect to Dask on that deployment. I was hoping not to have to hardcode the address in the python flow. I will take a look at the integration page. Ideally, the agent could be configured to connect to a dask deployment so that the flow writer would not have to concern themselves with the details of how the agent uses the dask deployment.

Hi @John Lewis, I believe you should be able to configure it as an environment variable on kubernetes and then reference that environment variable in the flow although I have not tried that myself

Thanks @Kalise Richmond. I will give that a try. By the way, we work for SAS and we are evaluating Prefect.

Might be related https://github.com/PrefectHQ/prefect-dask/pull/41

@Andrew Huang, that is interesting. I am new to python and prefect, so feel free to correct me. In a k8s situation, is it not better to have these types of details in the agent configuration instead of the flow?

@flow(task_runner=DaskTaskRunner(
    cluster_class="dask_kubernetes.operator.kubecluster.kubecluster.KubeCluster",
    cluster_kwargs={
        "image": "my-docker-image-with-dask-and-prefect",
    },
    adapt_kwargs={"minimum": 1, "maximum": 1},
))

Hi John! My name is Joey and I lead our Product Advocate team here. We’re happy to provide additional support for your POC. Please DM me if it would be helpful to grab some time with one of our engineers. Thanks!