Hi. This question is more related to Dask, but probably someone can help me. Is it possible to use 2 different default distributed Clients connected to external clusters in parallel Prefect tasks?

You would need to separate it between flows since you need to provide dask address to the executor. But why two clusters?

Actually I want to run prefect tasks in LocalDaskExecutor and when user want to have some heavy computation he can create a cluster(with resource manager for example) and submit Dask code there

You may then spin up a Dask cluster on demand and decide based on parameter values how many nodes it should have - could have just 1 for small workloads

Problem is not in cluster creation, but using 2 default clients(thread scope) connected to different clusters.

Thanks for explaining more, I still didn't get what is the actual problem you're trying to solve that way. You mentioned you want to dynamically allocate more or fewer resources for your workloads - assuming what you mentioned would be possible, does it address this problem?

The executor creates one Client, but you can create another within your task code I think. I don’t think LocalDask has a client. It uses a thread Pool. So the resource manager with LocalDask should work (I think). What error do you run into?

I have 2 parallel tasks and want them to work with different default clients. When I create Client(cluster or address) - it creates global client and use it for both tasks as default for Dask collections.

I see why it can be tricky after this. If it’s too tricky, you could just start a subflow maybe?

Thanks, will think about it.