Hello folks, while converting an old and monolithic ETL to a cool Prefect ETL, I stumbled upon the following situation:

@task("A small task"))
def small_task(some_params):
    # stuff happens here
    return something

@task("A complex task with many steps")
def complex_task(lots_of_params):
    # stuff happens
    # more stuff happens
    small_task_result = small_task(some_params)
    # even more stuff happens which uses small_task_result

My question is: what happens if I nest tasks as above? I suspect prefect would not like it, but I haven't set up a test infrastructure yet. Of course, I could use small_task as a simple function (remove the @task decorator), or break up complex_task in smaller tasks, but both solutions would mean for me to either leave things as they are (awful) or rewrite a lot of code.

Hey @Paolo, if needed, tasks can be nested by using

task.run()

, this will just run the Python underneath but it’s not treated as a task in a sense that Prefect won’t keep track of the state of that specific action. If that’s fine, it likely doesn’t have to be a task and can just be a function also.

As I feared, no error but still I will have to break down the big task, or leave it as it is. Thanks mate, cheers.