Hello all. I'm part of a team evaluating Prefect for my employer. Is there any data available on the performance of various components with very large graphs? I have scenarios that could extend up to 10M nodes and would regularly run around 1M nodes.
c
Chris White
12/10/2020, 2:28 AM
Hi @Joseph! There is no public performance data like that at this time, but if you’d like to learn more you can email
<mailto:david@prefect.io|david@prefect.io>
to set up a call. We definitely have users who run large scale graphs and we’ve found that the results depend largely on execution infrastructure and how well you can tune it; we are also working closely with a group right now who runs massive scale and are partnering with them on optimizing the experience.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.