https://prefect.io logo
Title
s

Sandeep Aggarwal

06/17/2020, 12:28 PM
Is there any architecture diagram for prefect core. It can be really helpful when self hosting core server and tell how various components in system interact with each other. Currently I am facing issues where individual components stops working/responding due to various reasons (Majority of them being triggered by system resource constraints). The architecture diagram will help understand if it is possible to selectively scale individual components.
j

josh

06/17/2020, 12:58 PM
Hi @Sandeep Aggarwal I am not immediately aware of a diagram at the ready. We definitely should add one to the docs, if you want to open a docs ticket on the repo that would be great! There are a few resources that have been linked in this channel before which may give you some context:

https://youtu.be/FETN0iivZps

https://docs.google.com/presentation/d/1TfOsYmsjgbwXRkiItb2ZeTW_oYxXWAWKMtEnEFOyPiA/edit#slide=id.g7453fd8b20_0_129
a

An Hoang

06/17/2020, 1:25 PM
The Pycon 2020 slide really helped! Thanks @josh 🙂
🙂 1
s

Sandeep Aggarwal

06/17/2020, 3:18 PM
Hey @josh thanks for quick response and sharing the resources. I will take a look at them sometime tomorrow. Can you also help with the list of components that support more than one running instances. From what i have observed, agent and scheduler can't have multiple instances running (at least targeting same workflows), but what about graphql, apollo & hasura? (Excuse me if this sounds too lame as my knowledge with Graphql ecosystem is quite limited as of now.)
j

josh

06/17/2020, 3:20 PM
For the agent you can have multiple running but it isn’t really beneficial unless each agent has a different set of labels pulling different flows. Similar for the scheduler you probably wouldn’t need more than one instance. As for the others I’m pretty sure multiple instances would help but I’m going to @Zachary Hughes for his opinion 🙂
z

Zachary Hughes

06/17/2020, 3:24 PM
Definitely not a "lame" question! I'll echo what Josh said: having more than one agent and scheduler are unlikely to be super beneficial. There's absolutely no harm in horizontally scaling any of your Apollo, GQL, or Hasura pods though. You might need to make some tweaks depending on how you're deploying, but multiple copies should be good! 👍
s

Sandeep Aggarwal

06/17/2020, 3:41 PM
Thats really helpful. Thanks for your inputs @josh @Zachary Hughes
j

Javier Garcia

08/26/2020, 6:37 PM
@josh - I couldn't access the slide, it seems like the link is now private. I requested access but not sure who will see that
j

Javier Garcia

08/26/2020, 6:45 PM
same issue
j

josh

08/26/2020, 6:48 PM
Here’s a PDF copy 🙂
j

Javier Garcia

08/26/2020, 6:48 PM
thanks!