Thread
#prefect-community
    j

    Jai P

    5 months ago
    Hi there! Can you visualize a flow in prefect 2.0, similar to
    flow.visualize()
    in prefect 1.0 (not using radar)? If not, are there plans to add that functionality?
    Anna Geller

    Anna Geller

    5 months ago
    The flow visualization feature requires the construction of a DAG. Prefect 2.0 is more dynamic than that which is why a much easier way to visualize your flow is to run it. Since the requirement to register your flow first to see it in the UI (since the Orion API is always lurking in the background) no longer exists in Prefect 2.0, simply running the flow and starting Orion will give you the flow visualization directly in the UI. But if the problem you try to solve this way is that you want to first see the flow diagram (to ensure dependencies are set correctly) before moving to production, you could solve this problem using multiple environments and switching between those with profiles. Say you use a local Orion API as a sandbox/dev environment, and you use it to interactively run your flows and see the Radar view of it locally before moving to prod. Once you are ready with testing, you could then switch your CLI profile to point to Cloud 2.0 or to some production API version, and move to that environment. But we understand the problem and you’ll likely see some more formalized guidance on that in the near future.
    j

    Jai P

    5 months ago
    Got it got it, ok that makes sense, and matches my understanding then. I'll keep an eye out for documentation, thank you! As a follow up then (im guessing the answer is no, correct me if im wrong) we can't easily traverse the dependencies of a flow since there aren't necessarily Edges anymore, right? For reference I'm talking about something like this
    Anna Geller

    Anna Geller

    5 months ago
    we can't easily traverse the dependencies of a flow since there aren't necessarily Edges anymore, right?
    Can you explain what problem do you try to solve this way? When it comes to unit testing your flows and tasks, Prefect 2.0 makes testing actually much easier by not being restrictive about what should be in your flow (i.e., Prefect 1.0 only allows calling tasks and setting dependencies, while 2.0 allows running arbitrary code). So you can test any custom behavior you like, depending on what would you want to test. A good approach would be thinking about how would you test your workflow without Prefect
    for dependency tracking you could do something like this
    j

    Jai P

    5 months ago
    Basically the problem I'm thinking of would be if we wanted to have our own custom visualization layer that isn't radar in the Orion ui, so we could represent to <insert stakeholder here> the steps taken in a particular flow
    I think the testing example may have been too specific, I definitely agree that testing is easier in prefect 2 since you don't have to think about tasks and flows and all that
    Anna Geller

    Anna Geller

    5 months ago
    our own custom visualization layer that isn't radar
    Can you explain why? Curious to hear what did you find lacking there?
    j

    Jai P

    5 months ago
    Don't think it's something that is lacking necessarily, but I think we view radar as like an “internal” tool that our engineering teams could interact with, but we may want to visualize an entire flow to say “an entity is passing through this flow, and this is the current point in the flow that the entity has reached”
    This is pretty complex so as I'm talking through it I'm actually wondering if maybe this is the responsibility of another system…I'll talk to my team to see what they think based on the knowledge we have. Just want to emphasize, you have helped a lot this weekend with my questions, so thank you so much!
    Anna Geller

    Anna Geller

    5 months ago
    by an entity, do you mean, for example, a specific dataset, data warehouse table, or a pickled ML model?
    j

    Jai P

    5 months ago
    so we're trying to use prefect to essentially do an ETL...we generate data about a person in our system, and there are a discrete set of steps that are taken in order to generate that data. the interesting thing is some of the steps aren't known up front, so we can't construct a flow the way we need to in 1.0, we need to dynamically generate flows at runtime which makes 2.0 suited to that problem. the interesting thing is, because a flow likely cannot run end to end the first time (meaning, we realize we can't generate all the data about a person, so we request other data to be generated and essentially pause the flow), we also want a way to represent that the individual reached a step, and is currently at that step. to do that...we essentially want to be able to traverse the flow and figure out the (for lack of a better word) graph of dependencies, and then visualize that in a way that makes sense to our users
    A -> B (person waiting here) -> C -> E <- D
         |
         V
         E
    Anna Geller

    Anna Geller

    5 months ago
    Such an interesting use case. Thanks so much for sharing! I think the part which is confusing to you is the difference between a flow and a flow run. 1. Flow - your Prefect 2.0 flow can be dynamic, and every time it runs, it can have different tasks generated within a flow run 2. *Flow ru*n - is a representation of a specific flow run execution/computation. While the flow run from yesterday could only have tasks A and B, the flow run today may have additional steps collecting additional steps about this person Within your flow code, you could dump the final end result to e.g. some key-value store and with every new flow run you could read the current state (read the current key-value pair), update it based on new information/new tasks and dump it again (e.g. overwrite the same key with new value).
    j

    Jai P

    5 months ago
    gotcha gotcha, that is definitely neat! agree that we'd want a separate store that tracks state of individuals. it is definitely a useful feature to be able to have variable tasks through a particular flow and be able to pause execution. I'd guess this also means that theoretically, i could write my flow such that i don't need to re-run A again, i can just skip to B on a subsequent flow run because i loaded the state (though, i'd need some step prior to A to determine where the flow should start)
    Anna Geller

    Anna Geller

    5 months ago
    I guess you would need to add some simple conditional logic based on the values available in your dictionary? I think it's useful to think about how would you solve it without Prefect, i.e. in plain Python? e.g. you could do:
    from prefect import task, flow
    from prefect import get_run_logger
    
    
    @task
    def get_value_for_key_from_redis() -> dict:
        return {"name": "Marvin"}
    
    
    @task
    def process_personal_data_step_a(user: dict):
        logger = get_run_logger()
        if user.get("A") is not None:
            <http://logger.info|logger.info>("processing step A")
            # do some action here
        else:
            <http://logger.info|logger.info>("Step A is already done")
    
    
    @task
    def process_personal_data_step_b(user: dict):
        logger = get_run_logger()
        if user.get("B") is not None:
            <http://logger.info|logger.info>("processing step B")
            # do some action here e.g. adding age:
            user["age"] = 42
        else:
            <http://logger.info|logger.info>("Step B is already done")
        return user
    
    
    @task
    def load_final_key_back_to_redis(user_dict: dict) -> None:
        pass  # load this dict back to Redis
    
    
    @flow
    def hw():
        user_data = get_value_for_key_from_redis()
        step_a = process_personal_data_step_a(user_data)
        step_b = process_personal_data_step_b(step_a)