Hello! 👋 Thank you for the work that you’re doing. I’ve been testing prefect over the past week and stumbled upon the topic of resource management. Created a github ticket for it; https://github.com/PrefectHQ/prefect/discussions/4700 Please review 🙇

you might find this thread on dask distributed to be insightful https://github.com/dask/distributed/issues/2118

I figured that as well but the flow of flows syntax isn’t as straight-forward as the flows/task relationship. (cc @Verun Rahimtoola )

agreed - not sure if there is a simpler approach here, perhaps someone on the prefect/dask side will chime in

Hey @Alexander van Eck, I have seen people use this flow of flow setup like @Marwan Sarieddine is suggesting. They then configure each subflow with a different exectuor. If you don’t want to do that route, you probably want to use Resource Managers . You can spin up and down the cluster configuration you need. This was also the topic of the Dask Summit keynote where Joe Schmid from SymphonyRM said that they configured Dask clusters at runtime. He used the legacy DaskCloudProviderEnvironment. I think that flow is flows is really the way to do this now. Resource Manager is good though if you specifically are writing Dask code.

Very interesting! I’ll digest this on the weekend. thank you 🙇

The 2021 summit hasn’t been uploaded yet. It wasn’t explicitly mentioned, but I asked him directly 😅

ah, that makes more sense haha

Oh my bad. I just checked and this video is correct. That's the one I was referring to.