@Marvin Question on task dependency. Given a flow doesn't require explicit task dependency to be defined, when should wait_for be used? Is it required when the tasks are dependent on completion but don't use the return value from previous?

def flow():
   _ = taskA()  #create table
   taskB()  #insert into table

If taskA requires taskB to complete first, should taskB use

wait_for=[_]

Flows defined without the wait_for appear to be completing the tasks sequently anyway. Does the flow know to complete taskA before taskB regardless?

When creating tasks within a flow in Prefect, implicit dependencies are generally recognized by the order in which tasks are defined, and by passing outputs of one task into another. However, if you need to ensure that a particular task waits for another to finish without using its output value, the

wait_for

argument can be useful. In your example, if

taskA

is running some sort of setup (e.g., creating a table) that

taskB

relies on (e.g., inserting into the table), but

taskB

doesn't need the output of

taskA

, you could use

wait_for

to explicitly define this dependency:

python
def flow():
   task_a = taskA()  # create table
   taskB(wait_for=[task_a])  # insert into table

By specifying

wait_for=[task_a]

in

taskB

, you are making it clear to the Prefect scheduler that

taskB

should not commence until

taskA

has completed. This is particularly important if you have a flow that might run tasks in parallel — without

wait_for

, Prefect may not inherently know that

taskB

must wait for

taskA

to finish. Flows that appear to complete tasks sequentially may be doing so because the tasks do not have any parallel execution configured, or because of the order they are called in the flow definition. However, it is better practice to explicitly state dependencies when the execution order is crucial, to ensure the correct task execution order regardless of the flow configuration. For more details and examples on task dependencies in flows, you can refer to the Prefect documentation and the basic flow example provided.

Since these tasks are not async, then I don't need to explicitly define dependency? Am I understanding correctly?

tasks will run in the order you've called them unless you're submitting them to a task runner that enables concurrency or parallelism

🧠 thought so. Thanks for confirming KG

of course KG 🙂 here's a doc I wrote recently that talks about the scenarios where

wait_for

is useful, should you ever need it