Jackson Maxfield Brown

    Jackson Maxfield Brown

    2 years ago
    Weird question is there are way to attach a result object to a task during flow construction? The idea being I have a flow that has a parameter of
    local
    or
    remote
    and so I want tasks to store results in
    LocalResult
    or
    S3Result
    accordingly:
    python run_my_flow.py # tasks are created with LocalResult (default)
    python run_my_flow.py --remote  # tasks are created with S3Result
    Weird pseudo code:
    with Flow("my_flow") as flow:
        result_target = Parameter("result_target", default="local")
    
        if result_target == "local":
            result_target_for_task_a = LocalResult("a_results/")
            result_target_for_task_b = LocalResult("b_results/")
        else result_target == "remote":
            result_target_for_task_a = S3Result("<s3://hello-world/a_results/>")
            result_target_for_task_b = S3Result("<s3://hello-world/b_results/>")
    
        items = task_a(result_handler=result_target_for_task_a)
        task_b(result_handler=result_target_for_task_b)
    Chris White

    Chris White

    2 years ago
    Hey @Jackson Maxfield Brown - a few notes: - you can’t condition on the value of your Parameter, as that is a runtime value that isn’t known at flow build time (but you can condition on your own custom CLI flags) - you can certainly set the
    result
    attribute of your tasks directly, but note that if you also use the
    target
    keyword, you’ll need to make sure you consider this logic that setting the attribute directly avoids: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/core/task.py#L286-L297 Otherwise I think you should be good!
    ah one more thing
    if you are using the functional API, you can also use the special
    task_args
    keyword, e.g.,
    items = task_a(task_args={"result": result_handler=result_target_for_task_a})
    ^^ this is probably preferred, because it will ensure the
    __init__
    logic is run
    Jackson Maxfield Brown

    Jackson Maxfield Brown

    2 years ago
    Ahhh thanks! Will give this all a look through. Yea I really didn't mean to use
    Parameter
    as I knew it would be runtime determined but I didn't want to type out argparse stuff in pseudo-code haha
    This is all super useful so thanks again!