Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

there is currently no documentation on how to scale prefect on multiple hosts ? is this on purpose so people only use cloud ?

hi <@U01DYQ1EVQQ>, your probably interested in the DaskClusterExecutor. Running prefect with this, would allow you to run over multiple hosts. You'd need a dask cluster setup in order to do so.

but this is only to run flows from cli or am I missing something ? can it work with dask and the prefect UI ?

because I suppose scaling the UI and the components would also be required if you want to scale.

i think the UI you are thinking about is only available in Prefect Cloud. Dask has its own UI, which shows you the running of tasks, you can inspect the cluster etc

Hi <@U01DYQ1EVQQ> - the reason there is no documentation on how to scale Prefect Server on multiple hosts is because it is not built to scale on multiple hosts.  Prefect Server is a very different product from Prefect Cloud that is intended for the local monitoring of your Flow runs.  The ability to handle massive scale and network across multiple machines is a very hard problem that requires specialized infrastructure and that is the problem we built Prefect Cloud to solve

whats so special about the infra ? :smile:

I'm a bit disappointed. I spent 3 days looking into the product after reading that it scales and it has a nice UI and just found out that I cannot deploy in the company I work for because scaling does not work out of the box or is not supported.

_Anything_ involving infrastructure is both incredibly complicated but also every one does it differently; the _reason_ we built Server the way we did is so that is doesn’t require infrastructure to deploy (which of course puts limitations on its possible use).

Prefect Cloud does scale out of the box and is fully supported.

well, I wish I knew that before I start reading and deploying. Good luck with it. I hope you make the "incredibly complicated" infra simpler and provide a way for users to deploy it to their hosts.

FWIW we built Cloud so that you still have full control of your deployment infrastructure and the code and data therein: <https://medium.com/the-prefect-blog/the-prefect-hybrid-model-1b70c7fd296>

sounds to me like you are just trying to monetise your work.
Having a hybrid solution just makes infra (specially networking) more complicated so don't try to sell it as its all easier for the users. At least its not an option for me and I am a user so ...

Open source software doesn't run on fresh air. prefect gives you a lot for free, if you need more then it must be of some value to you. Its had a lot invested already, good o/s software projects need funding from something. Monetizing a o/s project is s good thing, it helps it become better and ensure it doesn't end up moth balled like many projects, because there is a high cost to maintenance, community management and new features.

ye, and the best way which is proven to work in order to monetise opensource is to offer support.