@Marvin I am running prefect with single flow. I am using LocalCluster from Dask and passing its address to the DaskTaskRunner. Now, I have multiple stages inside a flow. Each stage requires different cluster configuration. Is it possible that I create again LocalCluster and somehow pass new client address to the existing task runner or replacing the old client with the new one?

Oh, the joy of juggling multiple cluster configurations within the same flow. Who wouldn't love to switch out the LocalCluster addresses in the midst of a perfectly sane workflow? But yes, let me check the documentation to see whether this chaotic desire of yours is feasible with Prefect. First things first, which version of Prefect are you using? The 2.x version or the 3.x version?

@Marvin 2.19.8

Well, it seems like having multiple cluster configurations is as thrilling as it sounds. Here are the nuggets of wisdom from the documentation and issues: 1. DaskTaskRunner Parameters: From the official Prefect documentation, the

DaskTaskRunner

accepts an

address

parameter to connect to an existing Dask scheduler and other parameters like

cluster_class

,

cluster_kwargs

, etc., to customize your Dask cluster. However, it does not directly support dynamically changing the client address mid-flow. - Prefect Task Runners - Usage Guide 2. GitHub Issues: There isn't direct support for switching the Dask client address on-the-fly within a single flow. However, you can manage separate Dask clusters and configure your flows to connect to them as needed. The relevant GitHub issues also don't provide a ready-made solution but indicate potential workarounds or configurations. - HPC Machines and Job Schedulers Issue - Coiled Cluster Issue In conclusion: - You cannot directly change the Dask client address mid-flow. - You might need to architect your flow to segregate tasks according to the cluster requirements or handle custom connections programmatically. Do these glimmers of hope align with your ambitious endeavor? Or does your chaotic dream need more grounding in reality?