https://prefect.io logo
Title
s

Shruti Hande

10/18/2022, 5:58 AM
Hello, Trying to run multiple python scripts in prefect. My use case is that, each task should represent as an individual python script and these all scripts/ tasks can be combined to form a single flow . HOW IS THIS ACHIEVABLE? Can we run bash commands as a single task in prefect version 2. #prefect-contributors #prefect-community #prefect-dbt
πŸ™Œ 2
βœ… 1
s

Santhosh Solomon (Fluffy)

10/18/2022, 10:28 AM
Yes, you can execute bash commands as tasks. But prefect will not have control over the task which has been instantiated over bash.
πŸ‘ 1
j

Jeff Hale

10/18/2022, 1:45 PM
pip install prefect-shell
to install the prefect-shell collection. Then you can do something like this:
from prefect import flow, task
from prefect_shell import shell_run_command


@flow
def example_shell_run_command_flow():
    return shell_run_command(
        command="touch new_file.py; ls",
        return_all=True,
    )


print(example_shell_run_command_flow())
shell_run_command
creates a prefect task. You can call each Python script consecutively in a single flow OR create a flow for each Python script (subflows). Here’s an example that uses subflows - you would just call your script where I create and list files:
from prefect import flow, task
from prefect_shell import shell_run_command


@flow
def first_shell_command():
    return shell_run_command(command="touch new_file1.py; ls ", return_all=True)


@flow
def second_shell_command():
    return shell_run_command(command="touch new_file2.py; ls", return_all=True)


@flow
def example_shell_run_command_flow():
    file1 = first_shell_command()
    file2 = second_shell_command()
    return file1 + file2


print(example_shell_run_command_flow())
Just note that because
shell_run_command
returns a task, you can’t use it within a task.
πŸ‘ 1
🍻 1
k

Khuyen Tran

10/18/2022, 3:27 PM
You can definitely have one task per file then just import all of the tasks in another file and use them inside a flow. However, it is recommended that you keep your tasks small for debugging and maintaining. If what you are looking for is a way to organize a code logic, using subflow as @Jeff Hale recommended is optimal
πŸ‘ 1
s

Shruti Hande

10/19/2022, 7:13 AM
Thanks for the insightful information @Santhosh Solomon (Fluffy) πŸ˜€
Thank you for this solution@Jeff Hale . It is of great help for me.! πŸ™‚πŸ™ŒπŸ’―
@Khuyen Tran Thank you for the solution, it is helpful. πŸ˜€πŸ’―