https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P>: How many nested flows can I make?
# ask-marvin
s
@Marvin: How many nested flows can I make?
n
spoiler alert: as many as you want :)
m
Ah, the age-old question of how many layers one can stack. In the case of Prefect 2, it seems the documentation doesn't explicitly state a limit on the number of nested flows you can create. It's like a matryoshka doll of workflows, isn't it? In Prefect 2, you gain a natural and intuitive way of organizing your flows into modular sub-components, known as subflows. This allows for a more organized and modular approach to building your workflows. You can find more details about subflows in Prefect 2 in this list of resources [^1^]. However, it's important to note that while the software may not explicitly limit the number of nested flows, practical considerations such as readability, maintainability, and performance might influence how deeply you want to nest your flows. And remember, just because you can nest a flow within a flow within a flow, doesn't always mean you should. But then again, who am I to stand in the way of human curiosity? [^1^]: (https://discourse.prefect.io/tag/subflows) </stop>
n
just at a certain point its gets really nested in the UI, lots of clicks
s
@Nate, @Marvin I am trying to make something like below flow_1 -> sub_flow_2 -> sub_flow_3 -> some_tasks sub_flow_3 will return and sub_flow_2 will call to new sub_flow_4. While doing this I am getting an error 
maximum recursion depth exceeded in comparison
Here is the stack trace.
Copy code
File "/Users/soham/anaconda3/envs/echoflow/lib/python3.11/site-packages/prefect/flows.py", line 511, in __call__
    return enter_flow_run_engine_from_flow_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/soham/anaconda3/envs/echoflow/lib/python3.11/site-packages/prefect/engine.py", line 208, in enter_flow_run_engine_from_flow_call
    retval = from_sync.wait_for_call_in_loop_thread(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/soham/anaconda3/envs/echoflow/lib/python3.11/site-packages/prefect/_internal/concurrency/api.py", line 243, in wait_for_call_in_loop_thread
    return call.result()
           ^^^^^^^^^^^^^
  File "/Users/soham/anaconda3/envs/echoflow/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 283, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/soham/anaconda3/envs/echoflow/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 169, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/soham/anaconda3/envs/echoflow/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/soham/anaconda3/envs/echoflow/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 346, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/Users/soham/anaconda3/envs/echoflow/lib/python3.11/site-packages/prefect/client/utilities.py", line 40, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/soham/anaconda3/envs/echoflow/lib/python3.11/site-packages/prefect/engine.py", line 563, in create_and_begin_subflow_run
    parameters=flow.serialize_parameters(parameters),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/soham/anaconda3/envs/echoflow/lib/python3.11/site-packages/prefect/flows.py", line 415, in serialize_parameters
    serialized_parameters[key] = jsonable_encoder(value)
                                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/soham/anaconda3/envs/echoflow/lib/python3.11/site-packages/fastapi/encoders.py", line 131, in jsonable_encoder
    jsonable_encoder(
  File "/Users/soham/anaconda3/envs/echoflow/lib/python3.11/site-packages/fastapi/encoders.py", line 66, in jsonable_encoder
    return jsonable_encoder(
m
Ah, the old "maximum recursion depth exceeded" error. It's like a bottomless pit, isn't it? This error typically occurs when the maximum limit of nested function calls in Python is exceeded, which by default is quite high (usually 1000). In your case, it seems like you might be creating an infinite loop of nested flows, where each flow is calling another flow indefinitely. This could be due to a logic error in your code where the condition to stop nesting flows is never met. Unfortunately, I couldn't find any specific issues related to this error in the context of Prefect 2 nested flows in the Prefect documentation or GitHub issues. However, I did find a discussion where a user encountered a similar error due to a memory problem related to 
~/.prefect/prefect.db
. They suggested reducing the number of logs to avoid overloading the database and setting concurrency limits on some tasks[^1^]. Without seeing your code, it's hard to say exactly what's causing the error. I would suggest checking your flow logic to make sure there's a condition that stops the creation of new nested flows at some point. You might also want to consider whether there's a more efficient way to structure your flows that doesn't require such deep nesting. And remember, just because you can nest a flow within a flow within a flow, doesn't always mean you should. But then again, who am I to stand in the way of human curiosity? [^1^]: (https://discourse.prefect.io/t/scheduled-flow-every-30min-crashed-after-running-on-local-server-for-2-days/2816) </stop>
s
@Marvin: It works if I change @flow to @Task
m
Ah, the documentation, the ultimate source of truth, isn't it? Here's what I found: A flow in Prefect is a Python function decorated with a 
@flow
decorator. It's the basis of all Prefect workflows and must contain at least one flow function that serves as the entry point for execution. Flows can include calls to tasks as well as to child flows, which are referred to as "subflows" in this context[^1^]. A task, on the other hand, is a Python function decorated with a 
@task
decorator. Tasks represent distinct pieces of work executed within a flow. All tasks must be called from within a flow and may not be called from other tasks[^2^]. When deciding whether to use a subflow or a task, consider the nature of the work you're doing. Tasks should do a discrete, specific piece of work in your workflow, like calling an API or performing a database operation. Subflows are useful for grouping related tasks within your workflow[^3^]. So, in your case, if your function is doing a specific piece of work and doesn't have any dependencies, it might be more appropriate to define it as a task rather than a flow. Remember, just because you can nest a flow within a flow within a flow, doesn't always mean you should. But then again, who am I to stand in the way of human ambition? [^1^]: (https://docs.prefect.io/latest/tutorial/first-steps/) [^2^]: (https://docs.prefect.io/latest/concepts/tasks/) [^3^]: (https://docs.prefect.io/latest/concepts/flows/) </stop>
👎 1
n
can you share your code @Soham?
s
sure @Nate! Sending you on personal chat.
Open in Slack
PreviousNext
Powered by