Hi
I am facing an issue
I need to add a shell task before my python tasks in a flow
but its not working , throwing some error like flow outside cant be called but when i comment out all task except the shell task it works
can I combine both shell task and python task in a single flow?
any example??
k
Kevin Kho
02/02/2022, 2:57 PM
Hi @shijas km,you have two options. First as the first task of the Flow
Copy code
shell = ShellTask(...)
with Flow(...) as flow:
s = shell()
something_else(upstream_tasks=[s])
if you really need it before the flow begins, you need to use it in a State Handler . This is not a task though. It’s just like a Python function.
Copy code
def mystatehandler(obj,old_state,new_state):
if new_state.is_running():
ShellTask().run(...)
with Flow(..., state_handlers=[mystatehandler]) as flow:
....
s
shijas km
02/02/2022, 2:59 PM
so what is this something__else()
n
Nate
02/02/2022, 2:59 PM
@shijas km hi! here's a very basic but functional example using both
ShellTask
and normal prefect tasks
Copy code
from prefect import Flow, task
from prefect.tasks.shell import ShellTask
@task(log_stdout=True)
def regular_task():
print('hi')
my_shell_task = ShellTask(
helper_script="cd ~",
)
with Flow("My Flow") as flow:
# task will be executed in home directory
contents = my_shell_task(command='ls')
regular_task()
if __name__ == "__main__":
flow.run()
s
shijas km
02/02/2022, 2:59 PM
where do we run flow.run()
k
Kevin Kho
02/02/2022, 3:00 PM
`something_else`would be another task but Nate has a full example for you
s
shijas km
02/02/2022, 3:23 PM
@Nate but I need the shell comand to run first , is this possible?
k
Kevin Kho
02/02/2022, 3:25 PM
You can set it as the upstream task or put it in the state handler like i outlined earlier
s
shijas km
02/02/2022, 3:32 PM
@Kevin Kho ok to make it simple
if I call shell task first and then call the other task then it will run in that order rite
is that fine
k
Kevin Kho
02/02/2022, 3:57 PM
Order is not guaranteed unless you set it as an explicit upstream like my first code snippet above
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.