I'm getting errors whenever I try pushing a prefect flow that's about 1500 tasks big. Is there a reason this happens?

Hi Diego! Could you share the error message(s) you're seeing? And just to clarify, you're hitting these errors when registering the flow, correct?

When I try to register, I get a very big dump of what I think is Haskell coming from Hasura. It takes around ~30 minutes to just print the whole dump.

Hmmm 🤔 Could you share a portion of the Haskell output? I'm not familiar with Haskell, but maybe some of the logs will help.

On a flow with ~2000 tasks, we get the following start (it does not materially change before my terminal crashes)

*** prefect.utilities.exceptions.ClientError: [{'message': '[{\'extensions\': {\'internal\': {\'statement\': \'WITH "edge__mutation_result_alias" AS (INSERT INTO "public"."edge" ( "mapped", "created", "flow_id", "tenant_id", "upstream_task_id", "key", "id", "updated", "downstream_task_id" ) VALUES (($2)::boolean, DEFAULT, ($1)::uuid, ($3)::uuid, ($4)::uuid, ($5)::text, DEFAULT, DEFAULT, ($6)::uuid), (($8)::boolean, DEFAULT, ($7)::uuid, ($9)::uuid, ($10)::uuid, ($11)::text, DEFAULT, DEFAULT, ($12)::uuid), (($14)::boolean, DEFAULT, ($13)::uuid, ($15)::uuid, ($16)::uuid, ($17)::text, DEFAULT, DEFAULT, ($18)::uuid), (($20)::boolean, DEFAULT, ($19)::uuid, ($21)::uuid, ($22)::uuid, ($23)::text, DEFAULT, DEFAULT, ($24)::uuid), (($26)::boolean, DEFAULT, ($25)::uuid, ($27)::uuid, ($28)::uuid, ($29)::text, DEFAULT, DEFAULT, ($30)::uuid), (($32)::boolean, DEFAULT, ($31)::uuid, ($33)::uuid, ($34)::uuid, ($35)::text, DEFAULT, DEFAULT, ($36)::uuid), (($38)::boolean, DEFAULT, ($37)::uuid, ($39)::uuid, ($40)::uuid, ($41)::text, DEFAULT, DEFAULT, ($42)::uuid), (($44)::boolean, DEFAULT, ($43)::uuid, ($45)::uuid, ($46)::uuid, ($47)::text, DEFAULT, DEFAULT, ($48)::uuid), (($50)::boolean, DEFAULT, ($49)::uuid, ($51)::uuid, ($52)::uuid, ($53)::text, DEFAULT, DEFAULT, ($54)::uuid), (($56)::boolean, DEFAULT, ($55)::uuid, ($57)::uuid, ($58)::uuid, ($59)::text, DEFAULT, DEFAULT, ($60)::uuid), (($62)::boolean, DEFAULT, ($61)::uuid, ($63)::uuid, ($64)::uuid, ($65)::text, DEFAULT, DEFAULT, ($66)::uuid), (($68)::boolean, DEFAULT, ($67)::uuid, ($69)::uuid, ($70)::uuid, ($71)::text, DEFAULT, DEFAULT, ($72)::uuid), (($74)::boolean, DEFAULT, ($73)::uuid, ($75)::uuid, ($76)::uuid, ($77)::text, DEFAULT, DEFAULT, ($78)::uuid), (($80)::boolean, DEFAULT, ($79)::uuid, ($81)::uuid, ($82)::uuid, ($83)::text, DEFAULT, DEFAULT, ($84)::uuid), (($86)::boolean, DEFAULT, ($85)::uuid, ($87)::uuid, ($88)::uuid, ($89)::text, DEFAULT, DEFAULT, ($90)::uuid), (($92)::boolean, DEFAULT, ($91)::uuid, ($93)::uuid, ($94)::uuid, ($95)::text, DEFAULT, DEFAULT, ($96)::uuid), (($98)::boolean, DEFAULT, ($97)::uuid, ($99)::uuid, ($100)::uuid, ($101)::text, DEFAULT, DEFAULT, ($102)::uuid), (($104)::boolean, DEFAULT, ($103)::uuid, ($105)::uuid, ($106)::uuid, ($107)::text, DEFAULT, DEFAULT, ($108)::uuid), (($110)::boolean, DEFAULT, ($109)::uuid, ($111)::uuid, ($112)::uuid, ($113)::text, DEFAULT, DEFAULT, ($114)::uuid), (($116)::boolean, DEFAULT, ($115)::uuid, ($117)::uuid, ($118)::uuid, ($119)::text, DEFAULT, DEFAULT, ($120)::uuid), (($122)::boolean, DEFAULT, ($121)::uuid, ($123)::uuid, ($124)::uuid, ($125)::text, DEFAULT, DEFAULT, ($126)::uuid), (($128)::boolean, DEFAULT, ($127)::uuid, ($129)::uuid, ($130)::uuid, ($131)::text, DEFAULT, DEFAULT, ($132)::uuid), (($134)::boolean, DEFAULT, ($133)::uuid, ($135)::uuid, ($136)::uuid, ($137)::text, DEFAULT, DEFAULT, ($138)::uuid), (($140)::boolean, DEFAULT, ($139)::uuid, ($141)::uuid, ($142)::uuid, ($143)::text, DEFAULT, DEFAULT, ($144)::uuid), (($146)::boolean, DEFAULT, ($145)::uuid, ($147)::uuid, ($148)::uuid, ($149)::text, DEFAULT, DEFAULT, ($150)::uuid), (($152)::boolean, DEFAULT, ($151)::uuid, ($153)::uuid, ($154)::uuid, ($155)::text, DEFAULT, DEFAULT, ($156)::uuid), (($158)::boolean, DEFAULT, ($157)::uuid, ($159)::uuid, ($160)::uuid, ($161)::text, DEFAULT, DEFAULT, ($162)::uuid), (($164)::boolean, DEFAULT, ($163)::uuid, ($165)::uuid, ($166)::uuid, ($167)::text, DEFAULT, DEFAULT, ($168)::uuid), (($170)::boolean, DEFAULT, ($169)::uuid, ($171)::uuid, ($172)::uuid, ($173)::text, DEFAULT, DEFAULT, ($174)::uuid), (($176)::boolean, DEFAULT, ($175)::uuid, ($177)::uuid, ($178)::uuid, ($179)::text, DEFAULT, DEFAULT, ($180)::uuid), (($182)::boolean, DEFAULT, ($181)::uuid, ($183)::uuid, ($184)::uuid, ($185)::text, DEFAULT, DEFAULT, ($186)::uuid), (($188)::boolean, DEFAULT, ($187)::uuid, ($189)::uuid, ($190)::uuid, ($191)::text, DEFAULT, DEFAULT, ($192)::uuid), (($194)::boolean, DEFAULT, ($193)::uuid, ($195)::uuid, ($196)::uuid, ($197)::text, DEFAULT, DEFAULT, ($198)::uuid), (($200)::boolean, DEFAULT, ($199)::uuid, ($201)::uuid, ($202)::uuid, ($203)::text, DEFAULT, DEFAULT, ($204)::uuid), (($206)::boolean, DEFAULT, ($205)::uuid, ($207)::uuid, ($208)::uuid, ($209)::text, DEFAULT, DEFAULT, ($210)::uuid), (($212)::boolean, DEFAULT, ($211)::uuid, ($213)::uuid, ($214)::uuid, ($215)::text, DEFAULT, DEFAULT, ($216)::uuid), (($218)::boolean, DEFAULT, ($217)::uuid, ($219)::uuid, ($220)::uuid, ($221)::text, DEFAULT, DEFAULT, ($222)::uuid), (($224)::boolean, DEFAULT, ($223)::uuid, ($225)::uuid, ($226)::uuid, ($227)::text, DEFAULT, DEFAULT, ($228)::uuid), (($230)::boolean, DEFAULT, ($229)::uuid, ($231)::uuid, ($232)::uuid, ($233)::text, DEFAULT, DEFAULT, ($234)::uuid), (($236)::boolean, DEFAULT, ($235)::uuid, ($237)::uuid, ($238)::uuid, ($239)::text, DEFAULT, DEFAULT, ($240)::uuid), (($242)::boolean, DEFAULT, ($241)::uuid, ($243)::uuid, ($244)::uuid, ($245)::text, DEFAULT, DEFAULT, ($246)::uuid), (($248)::boolean, DEFAULT, ($247)::uuid, ($249)::uuid, ($250)::uuid, ($251)::text, DEFAULT, DEFAULT, ($252)::uuid), (($254)::boolean, DEFAULT, ($253)::uuid, ($255)::uuid, ($256)::uuid, ($257)::text, DEFAULT, DEFAULT, ($258)::uuid), (($260)::boolean, DEFAULT, ($259)::uuid, ($261)::uuid, ($262)::uuid, ($263)::text, DEFAULT, DEFAULT, ($264)::uuid), (($266)::boolean, DEFAULT, ($265)::uuid, ($267)::uuid, ($268)::uuid, ($269)::text, DEFAULT, DEFAULT, ($270)::uuid), (($272)::boolean, DEFAULT, ($271)::uuid, ($273)::uuid, ($274)::uuid, ($275)::text, DEFAULT, DEFAULT, ($276)::uuid), (($278)::boolean, DEFAULT, ($277)::uuid, ($279)::uuid, ($280)::uuid, ($281)::text, DEFAULT, DEFAULT, ($282)::uuid), (($284)::boolean, DEFAULT, ($283)::uuid, ($285)::uuid, ($286)::uuid, ($287)::text, DEFAULT, DEFAULT, ($288)::uuid), (($290)::boolean, DEFAULT, ($289)::uuid, ($291)::uuid, ($292)::uuid, ($293)::text, DEFAULT, DEFAULT, ($294)::uuid), (($296)::boolean, DEFAULT, ($295)::uuid, ($297)::uuid, ($298)::uuid, ($299)::text, DEFAULT, DEFAULT, ($300)::uuid), (($302)::boolean, DEFAULT, ($301)::uuid, ($303)::uuid, ($304)::uuid, ($305)::text, DEFAULT, DEFAULT, ($306)::uuid), (($308)::boolean, DEFAULT, ($307)::uuid, ($309)::uuid, ($310)::uuid, ($311)::text, DEFAULT, DEFAULT, ($312)::uuid), (($314)::boolean, DEFAULT, ($313)::uuid, ($315)::uuid, ($316)::uuid, ($317)::text, DEFAULT, DEFAULT, ($318)::uuid), (($320)::boolean, DEFAULT, ($31

Thanks for grabbing that info! It looks like an error is thrown when trying to write to graphql as part of registering the flow. And I suspect logging the full query in the error message is crashing your terminal. 😅

I seem to be able to reproduce it with

from prefect import task, Flow, Task


class MyTask(Task):
    def run(self):
        return "hello"

tasks = [MyTask() for i in range(3000)]
flow = Flow(name = 'testflow', tasks = tasks)
for i in range(3000):
    flow.set_dependencies(task = tasks[i], upstream_tasks= tasks[max(0, i - 10):i] )
    
print('registering')
flow.register(build=True, project_name='TestProject')

This is super helpful, thank you! I'll do a bit of testing on my end now

Thanks for checking this!

Hey Diego - we're working on an update to smooth out registration for flows with many dependencies. Hoping to have the fix out late this week or early next week. I'll keep you updated.

I'm using prefect server, tested against 0.14.9 and 0.14.12

Afternoon Diego - unfortunately the underlying issue here is a bit tricker than I originally thought, we're still working through it. I'll keep you updated as we work through it. If you can share a bit more of your flow code, it might be possible to tweak it to avoid creating so many dependencies

I don't know if this is new information but I just found out the problem is an underlying limit of postgres. More precisely, line 1409 in https://doxygen.postgresql.org/fe-exec_8c_source.html . Is it possible to build flows incrementally? Say, if I have a flow, just add a task to it?

Nice find! I was still trying to figure out where in the stack we were hitting the limit

Is it possible to build flow incrementally?

Not exactly, but there might be some ways to get around it. The first thing that comes to mind is mapping https://docs.prefect.io/core/concepts/mapping.html. Mapping creates tasks dynamically at run time instead of creating them all on registration.

I have nontrivial dependencies between the tasks i'm simulating, so I don't think I can move it to mapping

Quick update here - as a short term solution we're working through 1. An enhancement to Prefect core to speed up flow registration on the client side 2. An enhancement to Prefect Server to break up task and edge creation when creating a flow With these two changes, I'm able to register the example flow you provided (3,000 tasks, ~30,000 edges). Is your production flow similar in size to the example you provided?

That is amazing! Yeah, it is roughly the size of my production flow

Hi Diego - the latest releases of Core (0.14.15) and Server (2021.04.06) contain the changes described above. Can you try registering your flow using the upgraded versions? Flow registration will still take a bit of time due to size, but it should work. Let me know if I can help with any part of the process!