Thread
#prefect-community
    c

    Christian

    2 years ago
    Hi all. I just spotted that we now have a first implementation of GreatExpectations in GitHub HEAD! Great stuff... I try to run the example and wonder how to access the GE return json with the validation results?
    Jim Crist-Harif

    Jim Crist-Harif

    2 years ago
    Glad you're interested in the GE integration, you have @Laura Lorenz (she/her) to thank for that šŸ˜ƒ
    c

    Christian

    2 years ago
    I was just looking how to integrate GE into a prefect flow when I spotted the PR on github. Waited another 5h and it was merged šŸ˜‰ - but I'm a prefect noob and not quite sure how the taks/ flow objects work. I need to access the large JSON that is returned by GE after a validation
    Jim Crist-Harif

    Jim Crist-Harif

    2 years ago
    I'm not very familiar with GE. It looks like our builtin prefect task returns the output of
    DataContext.run_validation_operator
    .
    c

    Christian

    2 years ago
    I think I'm not really getting how I can can access the result value itself: This is from the example:
    with Flow("great expectations example flow") as flow:
        checkpoint_name = Parameter("checkpoint_name")
        validations = ge_task.map(checkpoint_name)
        print("Result:", validations.result)
    but the print returns None. Maybe that's due to the use of map() ?
    Jim Crist-Harif

    Jim Crist-Harif

    2 years ago
    Ah, nothing has run yet at that point. Prefect has two stages of use: ā€¢ Flow build. This is everything inside the
    with Flow(...)
    block. At this point no tasks have run, you're just describing the flow you'd like to run later. ā€¢ Flow run. This happens locally when you call
    flow.run
    , but can also happen if you register a flow to run later with either cloud or server. To access the results after a run, look at:
    state = flow.run()
    print(state.result[validations].result)
    If you want to add a task after the GE task to handle the output, you can add another task to the flow and pass the results directly:
    @task
    def handle_ge_output(result):
       # do your stuff here
    
    with Flow("great expectations example flow") as flow:
        checkpoint_name = Parameter("checkpoint_name")
        validations = ge_task.map(checkpoint_name)
        handle_ge_output.map(validations)
    c

    Christian

    2 years ago
    Dooooh, yes I just figured that out. Sorry for the noob mistake šŸ™„
    Jim Crist-Harif

    Jim Crist-Harif

    2 years ago
    No worries! Glad you figured it out šŸ˜ƒ
    c

    Christian

    2 years ago
    Thanks for the help. šŸ‘ Back to my experiments šŸ˜Ž
    ... now I'm only struggling to force result_format to be COMPLETE for all expectations ... not sure if this is possible though