https://prefect.io logo
Title
o

Oliver Mannion

07/23/2022, 7:56 AM
The latest version of Orion introduces
.submit()
, eg:
@flow
def greetings(names: List[str]) -> None:
    for name in names:
        say_hello.submit(name)
        say_goodbye.submit(name)
but the flow still works without using
.submit()
. What are the implications of not using
.submit()
?
โœ… 1
v

Vitaliy Kotik

07/23/2022, 8:08 AM
without .submit tasks are being executed sequentially like if you didnโ€™t decorate them with @task at all.
๐Ÿ’ฏ 1
with .submit tasks are being executed concurrently
๐Ÿ‘ 1
:thank-you: 1
a

Anna Geller

07/23/2022, 1:15 PM
@Oliver Mannion you can read more here https://discourse.prefect.io/t/what-is-the-default-taskrunner-executor/63#prefect-20-1
:gratitude-thank-you: 1
m

Matthias

07/23/2022, 5:40 PM
And what if you combine tasks with and without
submit
in a flow (if thatโ€™s even possible)?
a

Anna Geller

07/23/2022, 6:13 PM
yes, it should be, if not @Matthias please report back
m

Matthias

07/24/2022, 6:58 AM
So then, some tasks are send to the taskrunner and some are not? Sounds awesome!
a

Anna Geller

07/24/2022, 10:49 AM
Exactly ๐Ÿ’ฏ the idea is that task runner is not always needed. It's primarily needed when you want to move to more advanced use cases such as parallel execution with async, Dask or Ray, but this is not always required, so now only when you explicitly submit a task to a task runner, it will be considered for this type of execution, otherwise it will run sequentially directly in a flow run process
:thank-you: 2
m

Matthias

07/24/2022, 7:38 PM
Maybe stupid next question: in light of what you explained, what is then the purpose of the sequential task runner?
โž• 2
v

Vitaliy Kotik

07/24/2022, 7:49 PM
debugging maybe. a more lockstep environment.
a

Anna Geller

07/24/2022, 10:06 PM
what is then the purpose of the sequential task runner?
@Matthias I believe the purpose is: if you want to operate on state dependencies rather than returned data (and even exceptions) from tasks, and you still want to use sequential execution, you need Sequential Task Runner:
@task
def task_a():
    pass

@task
def task_b():
    pass

@task
def task_c():
    pass

@task
def task_d():
    pass

@flow(task_runner=SequentialTaskRunner())
def my_flow():
    a = task_a.submit()
    b = task_b.submit()
    # Wait for task_a and task_b to complete
    c = task_c.submit(wait_for=[a, b])
    # task_d will wait for task_c to complete
    # Note: If waiting for one task it must still be in a list.
    d = task_d(wait_for=[c])
๐Ÿ‘ 1
m

Matthias

07/25/2022, 7:51 AM
Thanks for the explanation! Will that be in the documentation somewhere?
a

Anna Geller

07/25/2022, 3:41 PM
100%, if not please ping us in the next week or two ๐Ÿ˜„
๐Ÿ™Œ 1
a

Andreas

07/26/2022, 4:25 PM
Personllay I find this .submit change a little annoying... One of the "selling" points of Prefect 2.0 was, as stated by the introduction to Prefect 2.0 article:
automatically parallelize your synchronous Python script with one of our concurrent executors (enabled by default)
This automatic and easy parallelization that we could automatically achieve for all the tasks in a flow is now lost in last beta versions... From now on we need to provide the .submit for each task if we want all of the tasks in our script to run in parallel and handled by Dask.
a

Anna Geller

07/27/2022, 7:26 AM
There are always trade-offs but it will get even more interesting and configurable - follow the #announcements
๐Ÿ‘ 1