Jackson Maxfield Brown
09/04/2020, 12:02 AMlocal or remote and so I want tasks to store results in LocalResult or S3Result accordingly:
python run_my_flow.py # tasks are created with LocalResult (default)
python run_my_flow.py --remote # tasks are created with S3Result
Weird pseudo code:
with Flow("my_flow") as flow:
result_target = Parameter("result_target", default="local")
if result_target == "local":
result_target_for_task_a = LocalResult("a_results/")
result_target_for_task_b = LocalResult("b_results/")
else result_target == "remote":
result_target_for_task_a = S3Result("<s3://hello-world/a_results/>")
result_target_for_task_b = S3Result("<s3://hello-world/b_results/>")
items = task_a(result_handler=result_target_for_task_a)
task_b(result_handler=result_target_for_task_b)Chris White
result attribute of your tasks directly, but note that if you also use the target keyword, you’ll need to make sure you consider this logic that setting the attribute directly avoids: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/core/task.py#L286-L297
Otherwise I think you should be good!Chris White
Chris White
task_args keyword, e.g.,
items = task_a(task_args={"result": result_handler=result_target_for_task_a})
^^ this is probably preferred, because it will ensure the __init__ logic is runJackson Maxfield Brown
09/04/2020, 1:06 AMParameter as I knew it would be runtime determined but I didn't want to type out argparse stuff in pseudo-code hahaJackson Maxfield Brown
09/04/2020, 1:06 AM