How many nested flows can I make

Question

< Marvin> How many nested flows can I make

Nate · Answer

spoiler alert as many as you want

Marvin · Answer

Ah the age old question of how many layers one can stack In the case of Prefect 2 it seems the documentation doesn t explicitly state a limit on the number of nested flows you can create It s like a matryoshka doll of workflows isn t it In Prefect 2 you gain a natural and intuitive way of organizing your flows into modular sub components known as subflows This allows for a more organized and modular approach to building your workflows You can find more details about subflows in Prefect 2 in this ^1^ However it s important to note that while the software may not explicitly limit the number of nested flows practical considerations such as readability maintainability and performance might influence how deeply you want to nest your flows And remember just because you can nest a flow within a flow within a flow doesn t always mean you should But then again who am I to stand in the way of human curiosity ^1^ lt stop gt

Nate · Answer

just at a certain point its gets really nested in the UI lots of clicks

Soham · Answer

< Nate> < Marvin> I am trying to make something like below flow 1 gt sub flow 2 gt sub flow 3 gt some tasks sub flow 3 will return and sub flow 2 will call to new sub flow 4 While doing this I am getting an error `maximum recursion depth exceeded in comparison`

Soham · Answer

Here is the stack trace ``` File Users soham anaconda3 envs echoflow lib python3 11 site packages prefect flows py line 511 in call return enter flow run engine from flow call ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users soham anaconda3 envs echoflow lib python3 11 site packages prefect engine py line 208 in enter flow run engine from flow call retval = from sync wait for call in loop thread ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users soham anaconda3 envs echoflow lib python3 11 site packages prefect internal concurrency api py line 243 in wait for call in loop thread return call result ^^^^^^^^^^^^^ File Users soham anaconda3 envs echoflow lib python3 11 site packages prefect internal concurrency calls py line 283 in result return self future result timeout=timeout ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users soham anaconda3 envs echoflow lib python3 11 site packages prefect internal concurrency calls py line 169 in result return self get result ^^^^^^^^^^^^^^^^^^^ File Users soham anaconda3 envs echoflow lib python3 11 concurrent futures base py line 401 in get result raise self exception File Users soham anaconda3 envs echoflow lib python3 11 site packages prefect internal concurrency calls py line 346 in run async result = await coro ^^^^^^^^^^ File Users soham anaconda3 envs echoflow lib python3 11 site packages prefect client utilities py line 40 in with injected client return await fn args kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^ File Users soham anaconda3 envs echoflow lib python3 11 site packages prefect engine py line 563 in create and begin subflow run parameters=flow serialize parameters parameters ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users soham anaconda3 envs echoflow lib python3 11 site packages prefect flows py line 415 in serialize parameters serialized parameters key = jsonable encoder value ^^^^^^^^^^^^^^^^^^^^^^^ File Users soham anaconda3 envs echoflow lib python3 11 site packages fastapi encoders py line 131 in jsonable encoder jsonable encoder File Users soham anaconda3 envs echoflow lib python3 11 site packages fastapi encoders py line 66 in jsonable encoder return jsonable encoder ```

Marvin · Answer

Ah the old maximum recursion depth exceeded error It s like a bottomless pit isn t it This error typically occurs when the maximum limit of nested function calls in Python is exceeded which by default is quite high usually 1000 In your case it seems like you might be creating an infinite loop of nested flows where each flow is calling another flow indefinitely This could be due to a logic error in your code where the condition to stop nesting flows is never met Unfortunately I couldn t find any specific issues related to this error in the context of Prefect 2 nested flows in the Prefect documentation or GitHub issues However I did find a where a user encountered a similar error due to a memory problem related to `~ prefect prefect db` They suggested reducing the number of logs to avoid overloading the database and setting concurrency limits on some tasks ^1^ Without seeing your code it s hard to say exactly what s causing the error I would suggest checking your flow logic to make sure there s a condition that stops the creation of new nested flows at some point You might also want to consider whether there s a more efficient way to structure your flows that doesn t require such deep nesting And remember just because you can nest a flow within a flow within a flow doesn t always mean you should But then again who am I to stand in the way of human curiosity ^1^ lt stop gt

Soham · Answer

< Marvin> It works if I change flow to Task

Marvin · Answer

Ah the documentation the ultimate source of truth isn t it Here s what I found A in Prefect is a Python function decorated with a ` flow` decorator It s the basis of all Prefect workflows and must contain at least one flow function that serves as the entry point for execution Flows can include calls to tasks as well as to child flows which are referred to as subflows in this context ^1^ A on the other hand is a Python function decorated with a ` task` decorator Tasks represent distinct pieces of work executed within a flow All tasks must be called from within a flow and may not be called from other tasks ^2^ When deciding whether to use a subflow or a task consider the nature of the work you re doing Tasks should do a discrete specific piece of work in your workflow like calling an API or performing a database operation Subflows are useful for grouping related tasks within your workflow ^3^ So in your case if your function is doing a specific piece of work and doesn t have any dependencies it might be more appropriate to define it as a task rather than a flow Remember just because you can nest a flow within a flow within a flow doesn t always mean you should But then again who am I to stand in the way of human ambition ^1^ ^2^ ^3^ lt stop gt

Nate · Answer

can you share your code < Soham>

Soham · Answer

sure < Nate> Sending you on personal chat