Looking at this: <https://github.com/PrefectHQ/pre...
# ask-community
y
Looking at this: https://github.com/PrefectHQ/prefect/issues/3375 I'm trying to get ShellTask to run powershell. I have the powershell executable on the path in the machine, as "pwsh".
run_in_pwsh = ShellTask(name="Powershell run", shell="pwsh")
this is what I was trying to run. Then within the flow I had:
run_in_pwsh(command='ls')
The documentation for the shell argument says 'shell to run the command with; defaults to "bash"'. I assumed it would accept whatever as long as that kicked off in the terminal correctly? What I'm actually trying to do is run an azure command line utility called 'azcopy' (which is not part of the generic az cli). I need to do it both on a windows machine and a linux machine. Having separate scripts is fine. Any thoughts for how I could/should do this on windows?
k
We only support bash on Windows from this closed issue
y
Gotcha, and for my case of running another command line executable would this be the recommended way of doing it?
k
Yes it would still be the shell task as long as that tool is installed in your execution environment
šŸ™ 1
y
@Kevin Kho Hey I'm still running into issues after following along https://github.com/PrefectHQ/prefect/pull/4397 and https://github.com/PrefectHQ/prefect/issues/4393 I have wsl2, and I have C:\Windows\System32 where the bash.exe is added both to user and system paths. I've tried this:
Copy code
winshell = ShellTask(shell=True)
with Flow("ShellAzCopy") as flow:
    winshell(command='ls')
flow.run()
and
Copy code
winshell = ShellTask(shell=r"C:\\Windows\\System32\\bash.exe")
with Flow("ShellAzCopy") as flow:
    winshell(command='ls')
flow.run()
k
That sounds right to me. What is your error?
Did you try:
winshell = ShellTask(shell="bash")
y
Just did now also error. When shell=True I get this: [2022-01-14 102411-0800] INFO - prefect.TaskRunner | Task 'ShellTask': Starting task run... [2022-01-14 102411-0800] ERROR - prefect.TaskRunner | Task 'ShellTask': Exception encountered during task execution! Traceback (most recent call last): File "C:\Users\ykadmin081021\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\engine\task_runner.py", line 876, in get_task_run_state value = prefect.utilities.executors.run_task_with_timeout( File "C:\Users\ykadmin081021\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\utilities\executors.py", line 454, in run_task_with_timeout return task.run(*args, **kwargs) # type: ignore File "C:\Users\ykadmin081021\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\utilities\tasks.py", line 456, in method return run_method(self, *args, **kwargs) File "C:\Users\ykadmin081021\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\tasks\shell.py", line 133, in run with Popen( File "C:\Users\ykadmin081021\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "C:\Users\ykadmin081021\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 1360, in _execute_child args = list2cmdline(args) File "C:\Users\ykadmin081021\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 565, in list2cmdline for arg in map(os.fsdecode, seq): File "C:\Users\ykadmin081021\AppData\Local\Programs\Python\Python39\lib\os.py", line 822, in fsdecode filename = fspath(filename) # Does type-checking of
filename
. TypeError: expected str, bytes or os.PathLike object, not bool
When I do either "bash" or "C:\\Windows\\System32\\bash.exe" I get this: [2022-01-14 102504-0800] INFO - prefect.FlowRunner | Beginning Flow run for 'ShellAzCopy' [2022-01-14 102504-0800] INFO - prefect.TaskRunner | Task 'ShellTask': Starting task run... [2022-01-14 102504-0800] ERROR - prefect.ShellTask | Command failed with exit code 127 [2022-01-14 102504-0800] INFO - prefect.TaskRunner | FAIL signal raised: FAIL('Command failed with exit code 127') [2022-01-14 102505-0800] INFO - prefect.TaskRunner | Task 'ShellTask': Finished task run for task with final state: 'Failed' [2022-01-14 102505-0800] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
Someone had a comment in one of those issues about using the /mnt/c path, but is that necessary? it just gets very messy
k
I honestly don’t know cuz I dont use Windows
I went through it. I guess so.
y
Is it worth re-opening? The fix doesn't behave the way the fix suggests it should
Although my error isn't the same as that persons, that person was getting something about file not found
k
Let me ping Michael and see his thoughts, but response might be delayd
šŸ™ 1
z
Exit code 127 is an executable not found error, so it looks like it’s failing to find the shell.
We set
shell=True
on the Popen call if the platform is win32 to ensure the PATH is passed through. The tasks’s
shell
option isn’t the same as that.
What is
sys.platform
in wsl2?
y
>> sys.platform
'linux'
I see fair enough. Yeah I've tried giving it the full windows path to the bash executable
hmm when I use:
winshell = ShellTask(shell="/mnt/c/Windows/System32/bash.exe")
I get an exit code 1, even though I'm running my .py file from a terminal as admin
z
That may mean that it ran the shell correctly but the command failed
We haven’t tested on WSL at all though, so I don’t have any great suggestions for you here. Is bash not installed in WSL itself? Is there a reason you need to call out to a Windows installation of it?
y
the command I'm running is "ls", bash is installed by default when you install wsl
If this is what you mean:
winshell = ShellTask(shell="/bin/bash.exe")
this also returns 1
z
What if your command is an empty string
""
y
Still exit code 1, here's the whole flow+task:
Copy code
winshell = ShellTask(shell="")
with Flow("ShellAzCopy") as flow:
    winshell(command='ls')
flow.run()
z
command=ā€œā€, not shell
Interesting that it exited with a code of 1 still
y
This gives me 127:
Copy code
winshell = ShellTask(shell="bash")
with Flow("ShellAzCopy") as flow:
    winshell(command='')
flow.run()
This gives me 1:
Copy code
winshell = ShellTask(shell="/mnt/c/Windows/System32/bash.exe")
with Flow("ShellAzCopy") as flow:
    winshell(command='')
flow.run()
Please let me know if this issue gets re-opened or fixed, right now I've given up and I'm just using this fugly thing
Copy code
@task
def run_subprocess(cmd):
    completed = subprocess.run(["pwsh", "-Command", cmd])
    return completed

with Flow("ShellAzCopy") as flow:
    cmd = Parameter("cmd")
    flow.add_task(run_subprocess)
    run_subprocess.bind(cmd=cmd, flow=flow)
flow.run_config = LocalRun(
working_dir="C:\\Users\\ykadmin081021\\Documents\\Projects\\ToolsTesting\\prefect_trial\\shell_demo\\"
)
flow.run(cmd = ".\\azcopy_shell.ps1")
I just need something that works and have no choice on server os because server choice is based on the equipment they're using in the lab šŸ˜•
k
You don’t need to use add task inside the Flow and bind btw
Copy code
with Flow("ShellAzCopy") as flow:
    cmd = Parameter("cmd")
    run_subprocess(cmd=cmd)