Hey all! Is there a way to see all flows as a graph? Having a lot of dependencies between them (the dependencies are managed with StartFlowRun) I'd like to visually see them and trigger specific flows manually. I'm using Prefect Cloud.
k
Kevin Kho
06/25/2021, 7:23 PM
Hi @Nike Gurin-Petrovych, unfortunately, we don;t have this right now. See this for a few more details.
👀 1
n
Nike Gurin-Petrovych
06/25/2021, 7:24 PM
@Kevin Kho thanks, checking.
j
Jeremiah
06/25/2021, 9:13 PM
@Nike Gurin-Petrovych we’re absolutely going to be pursuing visualization enhancements for this in the future; the popularity of the orchestrator pattern caught us by surprise and we’re looking at ways to improve it!
n
Nike Gurin-Petrovych
06/26/2021, 8:47 AM
@Jeremiah would be so useful! Currently, it's a blocker for us to use Prefect instead of Airflow. To make it clear features that block us:
• to be able to see a full graph of all flows
• to be able to run a flow with all its ancestors. A typical use case - we update one of the data sources and we need to run all flows that use that data source or derived data sources
The following is inconvenient but we can work around it:
• register all flows in a folder including subfolders
• a simpler way to register an upstream flow. Currently code becomes too unreadable when I have to define a flow itself, then define another flow as StartRunFlow, then define StartRunFlow for each of the dependencies, and finally set another flow that actually contains the relationships. Instead of all the above would be great to have it as:
â—¦ define a flow itself
â—¦ set flow.set_upstream(a list of dependencies aka flows)
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.