Hi there! Can you visualize a flow in prefect 2.0, similar to

flow.visualize()

in prefect 1.0 (not using radar)? If not, are there plans to add that functionality?

The flow visualization feature requires the construction of a DAG. Prefect 2.0 is more dynamic than that which is why a much easier way to visualize your flow is to run it. Since the requirement to register your flow first to see it in the UI (since the Orion API is always lurking in the background) no longer exists in Prefect 2.0, simply running the flow and starting Orion will give you the flow visualization directly in the UI. But if the problem you try to solve this way is that you want to first see the flow diagram (to ensure dependencies are set correctly) before moving to production, you could solve this problem using multiple environments and switching between those with profiles. Say you use a local Orion API as a sandbox/dev environment, and you use it to interactively run your flows and see the Radar view of it locally before moving to prod. Once you are ready with testing, you could then switch your CLI profile to point to Cloud 2.0 or to some production API version, and move to that environment. But we understand the problem and you’ll likely see some more formalized guidance on that in the near future.

Got it got it, ok that makes sense, and matches my understanding then. I'll keep an eye out for documentation, thank you! As a follow up then (im guessing the answer is no, correct me if im wrong) we can't easily traverse the dependencies of a flow since there aren't necessarily Edges anymore, right? For reference I'm talking about something like this

we can't easily traverse the dependencies of a flow since there aren't necessarily Edges anymore, right?

Can you explain what problem do you try to solve this way? When it comes to unit testing your flows and tasks, Prefect 2.0 makes testing actually much easier by not being restrictive about what should be in your flow (i.e., Prefect 1.0 only allows calling tasks and setting dependencies, while 2.0 allows running arbitrary code). So you can test any custom behavior you like, depending on what would you want to test. A good approach would be thinking about how would you test your workflow without Prefect

Basically the problem I'm thinking of would be if we wanted to have our own custom visualization layer that isn't radar in the Orion ui, so we could represent to <insert stakeholder here> the steps taken in a particular flow

our own custom visualization layer that isn't radar

Can you explain why? Curious to hear what did you find lacking there?

Don't think it's something that is lacking necessarily, but I think we view radar as like an “internal” tool that our engineering teams could interact with, but we may want to visualize an entire flow to say “an entity is passing through this flow, and this is the current point in the flow that the entity has reached”

by an entity, do you mean, for example, a specific dataset, data warehouse table, or a pickled ML model?

so we're trying to use prefect to essentially do an ETL...we generate data about a person in our system, and there are a discrete set of steps that are taken in order to generate that data. the interesting thing is some of the steps aren't known up front, so we can't construct a flow the way we need to in 1.0, we need to dynamically generate flows at runtime which makes 2.0 suited to that problem. the interesting thing is, because a flow likely cannot run end to end the first time (meaning, we realize we can't generate all the data about a person, so we request other data to be generated and essentially pause the flow), we also want a way to represent that the individual reached a step, and is currently at that step. to do that...we essentially want to be able to traverse the flow and figure out the (for lack of a better word) graph of dependencies, and then visualize that in a way that makes sense to our users

A -> B (person waiting here) -> C -> E <- D
     |
     V
     E

Such an interesting use case. Thanks so much for sharing! I think the part which is confusing to you is the difference between a flow and a flow run. 1. Flow - your Prefect 2.0 flow can be dynamic, and every time it runs, it can have different tasks generated within a flow run 2. *Flow ru*n - is a representation of a specific flow run execution/computation. While the flow run from yesterday could only have tasks A and B, the flow run today may have additional steps collecting additional steps about this person Within your flow code, you could dump the final end result to e.g. some key-value store and with every new flow run you could read the current state (read the current key-value pair), update it based on new information/new tasks and dump it again (e.g. overwrite the same key with new value).

gotcha gotcha, that is definitely neat! agree that we'd want a separate store that tracks state of individuals. it is definitely a useful feature to be able to have variable tasks through a particular flow and be able to pause execution. I'd guess this also means that theoretically, i could write my flow such that i don't need to re-run A again, i can just skip to B on a subsequent flow run because i loaded the state (though, i'd need some step prior to A to determine where the flow should start)

I guess you would need to add some simple conditional logic based on the values available in your dictionary? I think it's useful to think about how would you solve it without Prefect, i.e. in plain Python? e.g. you could do:

from prefect import task, flow
from prefect import get_run_logger


@task
def get_value_for_key_from_redis() -> dict:
    return {"name": "Marvin"}


@task
def process_personal_data_step_a(user: dict):
    logger = get_run_logger()
    if user.get("A") is not None:
        <http://logger.info|logger.info>("processing step A")
        # do some action here
    else:
        <http://logger.info|logger.info>("Step A is already done")


@task
def process_personal_data_step_b(user: dict):
    logger = get_run_logger()
    if user.get("B") is not None:
        <http://logger.info|logger.info>("processing step B")
        # do some action here e.g. adding age:
        user["age"] = 42
    else:
        <http://logger.info|logger.info>("Step B is already done")
    return user


@task
def load_final_key_back_to_redis(user_dict: dict) -> None:
    pass  # load this dict back to Redis


@flow
def hw():
    user_data = get_value_for_key_from_redis()
    step_a = process_personal_data_step_a(user_data)
    step_b = process_personal_data_step_b(step_a)