I'm just trying to understand the prefect 2.0 architecture from a scaling perspective. I'm just trying to picture it in my mind..
I understand 3 of the components needed to self host are:
• Orion Server
• UI
• API Server
All these 3 components are part of Orion.
So to scale do we have multiple replicas start with
API Server HA:
Copy code
--scheduler --no-scheduler
│ --ui --no-ui
UI Server HA:
Copy code
--scheduler --no-scheduler
Or am I thing through this in the wrong fashion
Vishy ganesh
08/24/2022, 1:17 PM
I can see having multiple schedulers causing issues (unless prefect handles that similar to airflow)
c
Christopher Boyd
08/25/2022, 2:17 PM
Hi Vishy,
You can scale the front-end however you’d like - I believe it’s a ReplicaSet, which is fronted by a service, so UI / API can be scaled easily.
The agent, is entirely up to you - it can be one agent. or many agents with different tags.
The database is the harder piece to scale properly, because if you are self-hosting prefect server, your best option is to scale vertically
v
Vishy ganesh
08/25/2022, 3:20 PM
@Christopher Boyd I agree, the agents is upto us and that's a great feature with Prefect 🎉
So If I get you right, the scheduler part of Orion paired with the database SHOULD be scaled vertically as a guidance.
So technically if our DB is HA, we could let the scheduler failover to another scheduler (dormant) in case of DR needs
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.