https://prefect.io logo
Title
r

Ricky Zhang

01/10/2023, 8:44 PM
Our team is working on a PoC in Prefect 2. We deployed Prefect Orion and Agent using the helm chart from Github. We tried to hook up the existing Dask cluster in k8s with Prefect 2. But I can't find any doc about the Dask deployment setup other than specifying DaskTaskExecutor in the flow script. Should the Dask cluster hooking up done in the Agent?
k

Kalise Richmond

01/10/2023, 9:26 PM
Hi @Ricky Zhang, the agent needs to be able to communicate with the Dask cluster so as long as where ever the agent lives can communicate with the dask cluster it should be fine.
a

Andrew Huang

01/10/2023, 9:33 PM
I think DaskTaskExecutor is Prefect 1. For Prefect 2, there’s prefect-dask here https://prefecthq.github.io/prefect-dask/
j

John Lewis

01/10/2023, 10:52 PM
@Kalise Richmond, I work with Ricky. Can you point us to the best doc explaining how to get the agent to communicate with a Dask cluster (in k8s)?
@Andrew Huang, I work with Ricky. Are you saying the only way to use a k8s Dask cluster with Prefect is to create python code like this?
@flow(task_runner=DaskTaskRunner(address="<http://my-dask-cluster>"))
k

Kalise Richmond

01/10/2023, 11:12 PM
@John Lewis That looks correct. To run the tasks on the dask cluster, you would specify the DaskTaskRunner. https://docs.prefect.io/concepts/task-runners/#running-tasks-on-dask Andrew's link to the prefect-dask integration has a lot of examples as well. Is this not how you were expecting to use Dask with Prefect? If not, can you provide more details on what you are trying to do or what's not working?
j

John Lewis

01/10/2023, 11:56 PM
@Kalise Richmond, thanks for the reply. When using dask-kubernetes, you can set KUBECONFIG and it will connect to Dask on that deployment. I was hoping not to have to hardcode the address in the python flow. I will take a look at the integration page. Ideally, the agent could be configured to connect to a dask deployment so that the flow writer would not have to concern themselves with the details of how the agent uses the dask deployment.
k

Kalise Richmond

01/11/2023, 12:02 AM
Hi @John Lewis, I believe you should be able to configure it as an environment variable on kubernetes and then reference that environment variable in the flow although I have not tried that myself
j

John Lewis

01/11/2023, 12:10 AM
Thanks @Kalise Richmond. I will give that a try. By the way, we work for SAS and we are evaluating Prefect.
:marvin: 1
a

Andrew Huang

01/11/2023, 12:17 AM
j

John Lewis

01/11/2023, 12:26 AM
@Andrew Huang, that is interesting. I am new to python and prefect, so feel free to correct me. In a k8s situation, is it not better to have these types of details in the agent configuration instead of the flow?
@flow(task_runner=DaskTaskRunner(
    cluster_class="dask_kubernetes.operator.kubecluster.kubecluster.KubeCluster",
    cluster_kwargs={
        "image": "my-docker-image-with-dask-and-prefect",
    },
    adapt_kwargs={"minimum": 1, "maximum": 1},
))
j

Joseph Desjardins

01/11/2023, 1:14 AM
Hi John! My name is Joey and I lead our Product Advocate team here. We’re happy to provide additional support for your POC. Please DM me if it would be helpful to grab some time with one of our engineers. Thanks!