https://prefect.io logo
Title
j

Jackson Maxfield Brown

09/04/2020, 12:02 AM
Weird question is there are way to attach a result object to a task during flow construction? The idea being I have a flow that has a parameter of
local
or
remote
and so I want tasks to store results in
LocalResult
or
S3Result
accordingly:
python run_my_flow.py # tasks are created with LocalResult (default)
python run_my_flow.py --remote  # tasks are created with S3Result
Weird pseudo code:
with Flow("my_flow") as flow:
    result_target = Parameter("result_target", default="local")

    if result_target == "local":
        result_target_for_task_a = LocalResult("a_results/")
        result_target_for_task_b = LocalResult("b_results/")
    else result_target == "remote":
        result_target_for_task_a = S3Result("<s3://hello-world/a_results/>")
        result_target_for_task_b = S3Result("<s3://hello-world/b_results/>")

    items = task_a(result_handler=result_target_for_task_a)
    task_b(result_handler=result_target_for_task_b)
c

Chris White

09/04/2020, 12:39 AM
Hey @Jackson Maxfield Brown - a few notes: - you can’t condition on the value of your Parameter, as that is a runtime value that isn’t known at flow build time (but you can condition on your own custom CLI flags) - you can certainly set the
result
attribute of your tasks directly, but note that if you also use the
target
keyword, you’ll need to make sure you consider this logic that setting the attribute directly avoids: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/core/task.py#L286-L297 Otherwise I think you should be good!
ah one more thing
if you are using the functional API, you can also use the special
task_args
keyword, e.g.,
items = task_a(task_args={"result": result_handler=result_target_for_task_a})
^^ this is probably preferred, because it will ensure the
__init__
logic is run
j

Jackson Maxfield Brown

09/04/2020, 1:06 AM
Ahhh thanks! Will give this all a look through. Yea I really didn't mean to use
Parameter
as I knew it would be runtime determined but I didn't want to type out argparse stuff in pseudo-code haha
👍 1
This is all super useful so thanks again!