Hi @Marvin what is a subflow in prefect?

In Prefect, a subflow allows you to group related tasks within your workflow. It can be thought of as a nested flow or a child flow that is called inside the execution of another (parent) flow. Subflows help organize complex workflows into more manageable components and offer several advantages, such as: - Observability: Subflows have first-class observability within the Prefect UI and Prefect Cloud. They appear in the Flow Runs dashboard, making it easier to monitor their status at a glance. - Conditional flows: Subflows can be useful when you have a group of tasks that you want to run conditionally, based on some logic or state within the parent flow. - Reusability: By grouping tasks into subflows, you can easily reuse them across different parent flows. Subflows are a powerful feature when you need to create workflows with a degree of complexity and prefer to encapsulate logic in a structured and observable manner.

@Marvin When I query the DB for all flows, do subflow also return in this query?

When you run a flow in Prefect 2, whether it is tracked by the API server or executed ad hoc through a Python script, all flow runs, including subflows, are tracked. The Prefect UI provides debugging and observability for these flows, indicating that subflows are indeed included when you query the database for all flows. This means when you check for all flow runs, you can expect to see parent flows and their corresponding subflows which have been executed and tracked.

@Marvin If I want to query only for flows and not subflows, what query should I sent to prefect api server using REST API?

The information provided does not specifically detail how to query the Prefect REST API to retrieve only parent flows without subflows. Since the explicit REST API query details are not included in the excerpts, there seems to be no clear instruction in the documentation that directly answers this question. However, to accomplish this, you might need to look into the structure of the flow and subflow metadata and how they are stored in the Prefect API. It is possible that parent flows can be distinguished by a specific field or property that is not present in subflows. If such a distinctive field exists, you can use it to filter out subflows when making your REST API call. The exact query parameters would depend on the Prefect Core version and its API specification. If this specific filter is not immediately apparent from the API documentation, you may need to manually inspect the results of a general flow query and determine if there's a consistent attribute that differentiates parent flows from subflows. Alternatively, Prefect support or the Prefect community forums might provide guidance for constructing such a query, or a Prefect API release may introduce this functionality in the future.

@Marvin when I use this REST API: /api/flow_runs/count, does it request only parent flows?

The provided excerpts do not include specific information about the

/api/flow_runs/count

REST API endpoint and whether it differentiates between parent flow runs and subflow runs in its count. However, based on the behavior of subflow runs being treated like normal flow runs, as mentioned in the documentation, it is possible that the endpoint counts both parent flows and subflows together. If you require a count that is specific to only parent flows or subflows, you may need to look for or request additional filtering capabilities in the API or handle the differentiation in your own code after fetching the flow run data.

@Marvin is there a difference between flows and flow_runs?

In Prefect, there is a distinct difference between "flows" and "flow runs": - Flows: A flow is a construct that defines the workflow. It represents the sequence of tasks and the logic that orchestrates how these tasks are executed. Flows are defined using the

@flow

decorator in Python. They are templates or blueprints for the processes you want to run. A flow on its own is just a definition; it does not execute any logic until it is run. - Flow Runs: A flow run represents a single execution of a flow. Every time a flow is triggered to run, whether manually or through a scheduler, a flow run is created. Flow runs are instances of the flow that can be monitored and tracked. They capture the state of the workflow execution for observability. The Prefect API monitors flow runs, tracking each task's state within the flow and any subflows that might be called. In short, a flow defines what to do, and a flow run is an instance of doing it.