https://prefect.io logo
Title
v

Vadym Dytyniak

06/01/2022, 10:11 AM
Hi. This question is more related to Dask, but probably someone can help me. Is it possible to use 2 different default distributed Clients connected to external clusters in parallel Prefect tasks?
a

Anna Geller

06/01/2022, 10:16 AM
You would need to separate it between flows since you need to provide dask address to the executor. But why two clusters?
Already one distributed cluster seems hard to manage, curious why you need that
v

Vadym Dytyniak

06/01/2022, 10:18 AM
Actually I want to run prefect tasks in LocalDaskExecutor and when user want to have some heavy computation he can create a cluster(with resource manager for example) and submit Dask code there
a

Anna Geller

06/01/2022, 10:20 AM
You may then spin up a Dask cluster on demand and decide based on parameter values how many nodes it should have - could have just 1 for small workloads
v

Vadym Dytyniak

06/01/2022, 12:26 PM
Problem is not in cluster creation, but using 2 default clients(thread scope) connected to different clusters.
a

Anna Geller

06/01/2022, 12:57 PM
Thanks for explaining more, I still didn't get what is the actual problem you're trying to solve that way. You mentioned you want to dynamically allocate more or fewer resources for your workloads - assuming what you mentioned would be possible, does it address this problem?
k

Kevin Kho

06/01/2022, 10:30 PM
The executor creates one Client, but you can create another within your task code I think. I don’t think LocalDask has a client. It uses a thread Pool. So the resource manager with LocalDask should work (I think). What error do you run into?
v

Vadym Dytyniak

06/02/2022, 7:20 AM
I have 2 parallel tasks and want them to work with different default clients. When I create Client(cluster or address) - it creates global client and use it for both tasks as default for Dask collections.
k

Kevin Kho

06/02/2022, 2:54 PM
I see why it can be tricky after this. If it’s too tricky, you could just start a subflow maybe?
v

Vadym Dytyniak

06/02/2022, 3:31 PM
Thanks, will think about it.