I am running the server on premesis. If I create a very simple graph with 200 nodes and then click “Schematic” in the UI, the I get no page load and error code: 5. If I reduce the amount of nodes to 100, it seems to work just fine. Is this expected? 200 doesn’t feel like a lot nodes (we have dags with well over 100K nodes). Here’s my graph:

@task
    def return2(x):
        return 2 + x

    @task
    def say_hello(x):
        print(f"Hello, world! {x}")


    with Flow("My First Flow") as flow:
        for x in range(0, 100):
            say_hello(return2(x))

    flow.register("test")

Hi @Joseph - which version of Prefect Server are you running? There have been some improvements to schematics in recent months but this is definitely on our radar to continue improving

Thanks for the response. I’m running the latest server (I just installed it today). 100K nodes would include some mix of tasks and mapped instances of tasks. This is because we often map one set of tasks on one key and then shuffle that to a different set of tasks on other keys. I also have a very large and specialized execution environment wherein I handle cross task data transfer in a specific way to allow for these large graphs (in some cases the count is actually more like 3-5 million, but I’m ignoring those cases for now because of their scale).

Hi @Joseph, can you take a screenshot or paste the error you're seeing? There are some known issues with large schematics that we're working on but I haven't see that particular error before.

using chrome on a macbook pro

Oooh, the browser tab is crashing

It is in my first post to this thread.

Oh my, I completely missed that! Let me give that a try when I get a chance, sorry @Joseph!

Thanks!

I can reproduce the behavior but can't quite figure out why it's happening, 100 nodes isn't that many nodes so the worker that generates the schematic really shouldn't be running into memory constraints

I assume crashing the browser tab means that’s doing something chrome doesn’t like?

It could come from a few places, but I usually only see the schematic worker crash on large graphs (think 5 - 10 thousand edges)

Thanks for the extra info. The “schematic worker” is the backend that generates the schematic or a front end (in browser) component?

It's a browser thread that's allocated specifically to generate the schematic (to avoid locking the main thread)

Got it