Is there any architecture diagram for prefect core. It can be really helpful when self hosting core server and tell how various components in system interact with each other. Currently I am facing issues where individual components stops working/responding due to various reasons (Majority of them being triggered by system resource constraints). The architecture diagram will help understand if it is possible to selectively scale individual components.

Hi @Sandeep Aggarwal I am not immediately aware of a diagram at the ready. We definitely should add one to the docs, if you want to open a docs ticket on the repo that would be great! There are a few resources that have been linked in this channel before which may give you some context: https://docs.google.com/presentation/d/1TfOsYmsjgbwXRkiItb2ZeTW_oYxXWAWKMtEnEFOyPiA/edit#slide=id.g7453fd8b20_0_129

The Pycon 2020 slide really helped! Thanks @josh 🙂

Hey @josh thanks for quick response and sharing the resources. I will take a look at them sometime tomorrow. Can you also help with the list of components that support more than one running instances. From what i have observed, agent and scheduler can't have multiple instances running (at least targeting same workflows), but what about graphql, apollo & hasura? (Excuse me if this sounds too lame as my knowledge with Graphql ecosystem is quite limited as of now.)

For the agent you can have multiple running but it isn’t really beneficial unless each agent has a different set of labels pulling different flows. Similar for the scheduler you probably wouldn’t need more than one instance. As for the others I’m pretty sure multiple instances would help but I’m going to @Zachary Hughes for his opinion 🙂

Definitely not a "lame" question! I'll echo what Josh said: having more than one agent and scheduler are unlikely to be super beneficial. There's absolutely no harm in horizontally scaling any of your Apollo, GQL, or Hasura pods though. You might need to make some tweaks depending on how you're deploying, but multiple copies should be good! 👍

Thats really helpful. Thanks for your inputs @josh @Zachary Hughes

@josh - I couldn't access the slide, it seems like the link is now private. I requested access but not sure who will see that

@Javier Garcia try this https://docs.google.com/presentation/d/1TfOsYmsjgbwXRkiItb2ZeTW_oYxXWAWKMtEnEFOyPiA/edit?usp=sharing

same issue

Here’s a PDF copy 🙂

thanks!