Hello, Trying to run multiple python scripts in prefect. My use case is that, each task should represent as an individual python script and these all scripts/ tasks can be combined to form a single flow . HOW IS THIS ACHIEVABLE? Can we run bash commands as a single task in prefect version 2. #prefect-contributors #prefect-community #prefect-dbt

Yes, you can execute bash commands as tasks. But prefect will not have control over the task which has been instantiated over bash.

pip install prefect-shell

to install the prefect-shell collection. Then you can do something like this:

from prefect import flow, task
from prefect_shell import shell_run_command


@flow
def example_shell_run_command_flow():
    return shell_run_command(
        command="touch new_file.py; ls",
        return_all=True,
    )


print(example_shell_run_command_flow())

shell_run_command

creates a prefect task. You can call each Python script consecutively in a single flow OR create a flow for each Python script (subflows). Here’s an example that uses subflows - you would just call your script where I create and list files:

from prefect import flow, task
from prefect_shell import shell_run_command


@flow
def first_shell_command():
    return shell_run_command(command="touch new_file1.py; ls ", return_all=True)


@flow
def second_shell_command():
    return shell_run_command(command="touch new_file2.py; ls", return_all=True)


@flow
def example_shell_run_command_flow():
    file1 = first_shell_command()
    file2 = second_shell_command()
    return file1 + file2


print(example_shell_run_command_flow())

Just note that because

shell_run_command

returns a task, you can’t use it within a task.

You can definitely have one task per file then just import all of the tasks in another file and use them inside a flow. However, it is recommended that you keep your tasks small for debugging and maintaining. If what you are looking for is a way to organize a code logic, using subflow as @Jeff Hale recommended is optimal

Thanks for the insightful information @Santhosh Solomon (Fluffy) 😀