Quick question - if I have a set of tasks that don't depend on each other but that I want to run ser...

Jason Boorn

10/27/2021, 4:08 PM

Quick question - if I have a set of tasks that don't depend on each other but that I want to run serially, what's the correct way to do that?

Anna Geller

10/27/2021, 4:25 PM

If you want to ensure that one task runs after another in a specific order, then you need to set upstream or downstream dependencies, as shown here:

Copy code

from prefect import task, Flow


@task(log_stdout=True)
def hello_world():
    print("hello world")


@task(log_stdout=True)
def hello_world_2():
    print("hello world")

with Flow("basic-flow") as flow:
    t1 = hello_world()
    t2 = hello_world_2()
    t1.set_downstream(t2)

if __name__ == "__main__":
    flow.visualize()

Anna Geller

10/27/2021, 4:27 PM

but if the order is not important to you, you can just pass the list of tasks:

Copy code

from prefect import task, Flow


@task(log_stdout=True)
def hello_world():
    print("hello world")


@task(log_stdout=True)
def hello_world_2():
    print("hello world")

flow = Flow("basic-flow", tasks=[hello_world, hello_world_2])

if __name__ == "__main__":
    flow.visualize()

this should still run one after the other when using the default LocalExecutor, but I think this way you can’t guarantee that hello_world runs before hello_world_2. Does it make sense?

Jason Boorn

10/27/2021, 4:30 PM

Yup thats perfect - thanks

Kevin Kho

10/27/2021, 5:31 PM

I just wanna add another syntax:

Copy code

with Flow("basic-flow") as flow:
    t1 = hello_world()
    t2 = hello_world_2(upstream_tasks=[t1])

👍 1

3 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.