there is currently no documentation on how to scale prefect on multiple hosts ? is this on purpose so people only use cloud ?

hi @tsar, your probably interested in the DaskClusterExecutor. Running prefect with this, would allow you to run over multiple hosts. You'd need a dask cluster setup in order to do so.

but this is only to run flows from cli or am I missing something ? can it work with dask and the prefect UI ?

i think the UI you are thinking about is only available in Prefect Cloud. Dask has its own UI, which shows you the running of tasks, you can inspect the cluster etc

Hi @tsar - the reason there is no documentation on how to scale Prefect Server on multiple hosts is because it is not built to scale on multiple hosts. Prefect Server is a very different product from Prefect Cloud that is intended for the local monitoring of your Flow runs. The ability to handle massive scale and network across multiple machines is a very hard problem that requires specialized infrastructure and that is the problem we built Prefect Cloud to solve

whats so special about the infra ? 😄

Anything involving infrastructure is both incredibly complicated but also every one does it differently; the reason we built Server the way we did is so that is doesn’t require infrastructure to deploy (which of course puts limitations on its possible use). Prefect Cloud does scale out of the box and is fully supported.

well, I wish I knew that before I start reading and deploying. Good luck with it. I hope you make the "incredibly complicated" infra simpler and provide a way for users to deploy it to their hosts.

FWIW we built Cloud so that you still have full control of your deployment infrastructure and the code and data therein: https://medium.com/the-prefect-blog/the-prefect-hybrid-model-1b70c7fd296

sounds to me like you are just trying to monetise your work. Having a hybrid solution just makes infra (specially networking) more complicated so don't try to sell it as its all easier for the users. At least its not an option for me and I am a user so ...

Open source software doesn't run on fresh air. prefect gives you a lot for free, if you need more then it must be of some value to you. Its had a lot invested already, good o/s software projects need funding from something. Monetizing a o/s project is s good thing, it helps it become better and ensure it doesn't end up moth balled like many projects, because there is a high cost to maintenance, community management and new features.

ye, and the best way which is proven to work in order to monetise opensource is to offer support.