https://prefect.io logo
Title
j

Joseph

12/11/2020, 10:44 PM
I am running the server on premesis. If I create a very simple graph with 200 nodes and then click “Schematic” in the UI, the I get no page load and error code: 5. If I reduce the amount of nodes to 100, it seems to work just fine. Is this expected? 200 doesn’t feel like a lot nodes (we have dags with well over 100K nodes). Here’s my graph:
@task
    def return2(x):
        return 2 + x

    @task
    def say_hello(x):
        print(f"Hello, world! {x}")


    with Flow("My First Flow") as flow:
        for x in range(0, 100):
            say_hello(return2(x))

    flow.register("test")
n

nicholas

12/11/2020, 10:49 PM
Hi @Joseph - which version of Prefect Server are you running? There have been some improvements to schematics in recent months but this is definitely on our radar to continue improving
And to clarify - when you're talking about 100k nodes, that's discreet tasks or including things like large mapped pipelines?
j

Joseph

12/11/2020, 11:41 PM
Thanks for the response. I’m running the latest server (I just installed it today). 100K nodes would include some mix of tasks and mapped instances of tasks. This is because we often map one set of tasks on one key and then shuffle that to a different set of tasks on other keys. I also have a very large and specialized execution environment wherein I handle cross task data transfer in a specific way to allow for these large graphs (in some cases the count is actually more like 3-5 million, but I’m ignoring those cases for now because of their scale).
So to follow up is there any particular/known issues that are causing this or do I need to go debug it myself?
n

nicholas

12/14/2020, 3:10 PM
Hi @Joseph, can you take a screenshot or paste the error you're seeing? There are some known issues with large schematics that we're working on but I haven't see that particular error before.
j

Joseph

12/14/2020, 3:17 PM
using chrome on a macbook pro
Doubt that is all that helpful
And I don’t see anything in the server output in the terminal that looks interesting. Should I enable some sort of verbose logging or something like that?
n

nicholas

12/14/2020, 3:24 PM
Oooh, the browser tab is crashing
That's definitely helpful but not something I've seen before with schematics
Are you able to share the code that's leading to this? I'd like to open a ticket for us to look into the problem, I don't think there's much more you can do on your end without getting into some tricky memory snapshots
(a minimum reproduction would be best if possible)
j

Joseph

12/14/2020, 5:01 PM
It is in my first post to this thread.
And it’s about as minimum as any repro could be 🙂
n

nicholas

12/14/2020, 5:03 PM
Oh my, I completely missed that! Let me give that a try when I get a chance, sorry @Joseph!
j

Joseph

12/14/2020, 5:03 PM
Thanks!
Any luck?
n

nicholas

12/17/2020, 5:34 PM
I can reproduce the behavior but can't quite figure out why it's happening, 100 nodes isn't that many nodes so the worker that generates the schematic really shouldn't be running into memory constraints
j

Joseph

12/17/2020, 5:34 PM
I assume crashing the browser tab means that’s doing something chrome doesn’t like?
(I’m more of a systems than front end devleoper)
If I wanted to debug it how would I get setup for that? Is it just Chrome Developer tools?
n

nicholas

12/17/2020, 5:36 PM
It could come from a few places, but I usually only see the schematic worker crash on large graphs (think 5 - 10 thousand edges)
Yeah that's the best tool, you can also spin up a local dev server of the UI and test against either your Server or Prefect Cloud and add breakpoints in the form of
console.log
I won't have a chance to dive deeper into this issue this week but schematic improvements in general area on the radar for the next month or so
Yeah graphviz is handling this flow no problem, so this must be front-end specific
message has been deleted
j

Joseph

12/17/2020, 7:25 PM
Thanks for the extra info. The “schematic worker” is the backend that generates the schematic or a front end (in browser) component?
n

nicholas

12/17/2020, 7:29 PM
It's a browser thread that's allocated specifically to generate the schematic (to avoid locking the main thread)
j

Joseph

12/17/2020, 7:36 PM
Got it
default 12:12:40.328725-0800 chrome_crashpad_handler [1217/121240.328676⚠️process_memory_mac.cc(93)] mach_vm_read(0x7ffeea19c000, 0x2000): (os/kern) invalid address (1)
Looks like some sort of segfault in Chrome