Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi all! I'm looking into a single tool to unify data flows and Prefect looks interesting! From the installation docs I can see it's simply a `pip install` . I am curious what high-availability options there are in case the master/scheduling/coordinator(?) node fails, and where the configurations/flows are stored that one can restore from.

Hello <@U0146HD6Y1M>! Welcome to the Prefect Community. Most Prefect users will utilize a Kubernetes Agent on a cluster to implement a high-availability setup. There’s a great tutorial on scaling out with this particular agent you can read in the <https://docs.prefect.io/orchestration/tutorial/k8s.html|documentation>, as well as more information on that particular <https://docs.prefect.io/orchestration/agents/kubernetes.html|Agent>. Additionally, Prefect provides a variety of <https://docs.prefect.io/orchestration/execution/storage_options.html|storage options> for your flows including local storage, Azure, AWS, GCS, and Docker. I hope these resources get you going in the right direction!