<@ULVA73B9P> I have a flow called main and two sub...
# ask-marvin
w
@Marvin I have a flow called main and two subflows name work1 and work2 How can I see the logs from work1 and work2 on the main flow? Also, logs from work1 (even on the individual page) are not showing (display This run didn't generate logs) , which steps I can take to investigate whats is the issue?
m
To view the logs from your subflows
work1
and
work2
in the main flow, you need to understand that a subflow run is created when a flow function is called inside the execution of another flow. The primary flow is the "parent" flow. The flow created within the parent is the "child" flow or "subflow." Subflow runs behave like normal flow runs. There is a full representation of the flow run in the backend as if it had been called separately. When a subflow starts, it will create a new task runner for tasks within the subflow. When the subflow completes, the task runner is shut down. Subflows will block execution of the parent flow until completion. However, asynchronous subflows can be run in parallel by using AnyIO task groups or asyncio.gather. Subflows differ from normal flows in that they will resolve any passed task futures. More Info Regarding the issue of logs from
work1
not showing, it could be due to several reasons. One common issue is that the task-level logs are not being captured correctly. You can check the console where you ran the agent to see if the logs are being displayed there. If you see the logs in the console but not in the Prefect Cloud interface, it could indicate an issue with the logging configuration or the communication between your agent and Prefect Cloud. More Info
w
For any human reading this, the issue was that work1 was actually another flow with more subflows and the logs was there. But regarding the main issue (seeing logs from subflow on main flow) that still a huge downside =/
k
is there something you're specifically looking to get out of running subflows that you can't get from running the code as tasks or undecorated python functions? asking because you're not the first person I've seen wanting to have their subflow logs display inline with parent flow logs
w
My user case is that a subflow is raising a log.warning for something that affects when debugging the code, and in my opinion, this information should be available right on the frontend page (no one goes under the subsection to read the logs, in my humble opinion). Is there any workaround for this, or is it on the roadmap?
ah sorry I didn't read your question correctly
> specifically looking to get out of running subflows that you can't get from running the code as tasks We have "logic" breaks of the code. PreProcessing; Feature Enginering, FeatureEngineering tasks; Modelling, and so on. We divided that into subflows because, conceptually, they have different "concerns" for code, but everything goes to the same execution (evaluate this Experiment). if we putted that into a task (to separate the concerns) this task would not be able to call other utilities or necessary tasks (sub-tasks are not allowed, right?) About not keeping it as a normal function, we would want to run this subflows in parallel (future) Finally, we want everything related to Prefect (and pipelines) on an external layer and not mixed with our Python code. Perhaps we are yet with v1 mentality in our codebase; but we are trying to avoid flows/tasks code in the middle of our Python code.
n
> Finally, we want everything related to Prefect (and pipelines) on an external layer and not mixed with our Python code. > Perhaps we are yet with v1 mentality in our codebase; but we are trying to avoid flows/tasks code in the middle of our Python code not addressing the logs thing, but on this last point, a big difference with prefect 2 is that the graph is discovered at runtime, so there isnt a direct analogue to the imperative API where you build the graph up front by saying like
Flow.add_task(..., upstream_tasks=[...]
however, you're free to dynamically wrap vanilla python functions with
task
or
flow
like
Copy code
from my_pure_python import some_fn

@flow
def dynamic_flow():
   some_task = task(log_prints=True)(some_fn) # just set log_prints to show how to set options
   some_flow = flow(log_prints=True)(some_fn)

   task_result = some_task()