Jackson Maxfield Brown
09/04/2020, 12:02 AMlocal
or remote
and so I want tasks to store results in LocalResult
or S3Result
accordingly:
python run_my_flow.py # tasks are created with LocalResult (default)
python run_my_flow.py --remote # tasks are created with S3Result
Weird pseudo code:
with Flow("my_flow") as flow:
result_target = Parameter("result_target", default="local")
if result_target == "local":
result_target_for_task_a = LocalResult("a_results/")
result_target_for_task_b = LocalResult("b_results/")
else result_target == "remote":
result_target_for_task_a = S3Result("<s3://hello-world/a_results/>")
result_target_for_task_b = S3Result("<s3://hello-world/b_results/>")
items = task_a(result_handler=result_target_for_task_a)
task_b(result_handler=result_target_for_task_b)
Chris White
09/04/2020, 12:39 AMresult
attribute of your tasks directly, but note that if you also use the target
keyword, you’ll need to make sure you consider this logic that setting the attribute directly avoids: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/core/task.py#L286-L297
Otherwise I think you should be good!task_args
keyword, e.g.,
items = task_a(task_args={"result": result_handler=result_target_for_task_a})
^^ this is probably preferred, because it will ensure the __init__
logic is runJackson Maxfield Brown
09/04/2020, 1:06 AMParameter
as I knew it would be runtime determined but I didn't want to type out argparse stuff in pseudo-code haha