Oliver Mannion
07/23/2022, 7:56 AM.submit()
, eg:
@flow
def greetings(names: List[str]) -> None:
for name in names:
say_hello.submit(name)
say_goodbye.submit(name)
but the flow still works without using .submit()
. What are the implications of not using .submit()
?Vitaliy Kotik
07/23/2022, 8:08 AMAnna Geller
07/23/2022, 1:15 PMMatthias
07/23/2022, 5:40 PMsubmit
in a flow (if thatโs even possible)?Anna Geller
07/23/2022, 6:13 PMMatthias
07/24/2022, 6:58 AMAnna Geller
07/24/2022, 10:49 AMMatthias
07/24/2022, 7:38 PMVitaliy Kotik
07/24/2022, 7:49 PMAnna Geller
07/24/2022, 10:06 PMwhat is then the purpose of the sequential task runner?@Matthias I believe the purpose is: if you want to operate on state dependencies rather than returned data (and even exceptions) from tasks, and you still want to use sequential execution, you need Sequential Task Runner:
@task
def task_a():
pass
@task
def task_b():
pass
@task
def task_c():
pass
@task
def task_d():
pass
@flow(task_runner=SequentialTaskRunner())
def my_flow():
a = task_a.submit()
b = task_b.submit()
# Wait for task_a and task_b to complete
c = task_c.submit(wait_for=[a, b])
# task_d will wait for task_c to complete
# Note: If waiting for one task it must still be in a list.
d = task_d(wait_for=[c])
Matthias
07/25/2022, 7:51 AMAnna Geller
07/25/2022, 3:41 PMAndreas
07/26/2022, 4:25 PMautomatically parallelize your synchronous Python script with one of our concurrent executors (enabled by default)This automatic and easy parallelization that we could automatically achieve for all the tasks in a flow is now lost in last beta versions... From now on we need to provide the .submit for each task if we want all of the tasks in our script to run in parallel and handled by Dask.
Anna Geller
07/27/2022, 7:26 AM