Hi anyone has tried to use shelltask to run powershell scrip Prefect Community #ask-community

Join Slack

Hi, anyone has tried to use shelltask to run power...

# ask-community

David Yang

11/22/2021, 8:18 PM

Hi, anyone has tried to use shelltask to run powershell scripts?

Kevin Kho

11/22/2021, 8:21 PM

Hey @David Yang, maybe you can changing the shell from bash to powershell? There is a

shell

argument.

💯 1

upvote 1

David Yang

11/22/2021, 8:32 PM

yeah. it works after I fixed a typo.

👍 1

Akiva Wolberg

11/30/2021, 10:32 PM

@David Yang I was just trying to do this! Do you have an example of what worked? I'm getting odd errors when I set shell="powershell" or shell="powershell.exe" (windows asks what program to use to run it), and shell="cmd" hangs indefinitely.

Kevin Kho

12/01/2021, 12:34 AM

What errors do you get?

Kevin Kho

12/01/2021, 12:37 AM

Could you try doing this (without Prefect) and seeing if it works?

David Yang

12/01/2021, 2:13 PM

this is the my code: psobj = prefect.tasks.shell.ShellTask( shell="powershell") psobj.run(command="./dir.ps1",helper_script="cd C:\\Users\\yangd\\Documents") Make sure the path is correct.

upvote 1

Akiva Wolberg

12/01/2021, 8:33 PM

@Kevin Kho Thanks! I'm not actually running a powershell script, I'm using the R command line (in powershell or cmd, I don't really care which). I can't use bash because I have Windows Subsystem for Linux installed, which is associated with bash.exe, and I don't have R or any of the other relevant packages installed in WSL so therefore they're not available in bash.

Kevin Kho

12/01/2021, 8:34 PM

Ah I see

Akiva Wolberg

12/01/2021, 9:02 PM

OK, here's a minimal example: test.R:

Copy code

print("got to R")

r_shelltask.py:

Copy code

from prefect import Flow
from prefect.tasks.shell import ShellTask

rshell = ShellTask(
    shell = "cmd",
    helper_script="echo 'got to cmd'",
    stream_output=True,
)

with Flow("abc") as f:
    r_result = rshell(
        command = "Rscript test.r"
    )

f.run()

output on running this:

Copy code

PS C:\Users\lwolberg\Desktop\workingdata> py r_shelltask.py
[2021-12-01 16:01:48-0500] INFO - prefect.FlowRunner | Beginning Flow run for 'abc'
[2021-12-01 16:01:48-0500] INFO - prefect.TaskRunner | Task 'ShellTask': Starting task run...
[2021-12-01 16:01:48-0500] INFO - prefect.ShellTask | Microsoft Windows [Version 10.0.19043.1165]
[2021-12-01 16:01:48-0500] INFO - prefect.ShellTask | (c) Microsoft Corporation. All rights reserved.
[2021-12-01 16:01:48-0500] INFO - prefect.ShellTask |

And then it hangs indefinitely.

Akiva Wolberg

12/01/2021, 9:05 PM

I also tried this; "Command failed with exit code 2" and I can't find any documentation on what exit code 2 is.

Copy code

from prefect import Flow
from prefect.tasks.shell import ShellTask


rshell = ShellTask(
    shell = "Rscript",
    helper_script="echo 'got to cmd'",
    stream_output=True,
)

with Flow("abc") as f:
    r_result = rshell(
        command = "test.r"
    )

f.run()

Akiva Wolberg

12/01/2021, 9:07 PM

With

shell="powershell"

it appears to work (reports Task 'ShellTask' with final state 'Success'), but it doesn't print either the helper script or the R script, just offers to open a prefect temp file.

Kevin Kho

12/01/2021, 9:12 PM

Can you try:

Copy code

rshell = ShellTask(
    shell = "cmd",
    helper_script="echo 'got to cmd'",
    stream_output=True,
    return_all=True
)

The

return_all

specifically

Akiva Wolberg

12/01/2021, 9:25 PM

Same output; that also hangs indefinitely.

Kevin Kho

12/01/2021, 9:26 PM

Am a bit confused. Does echo really work on powershell?

Akiva Wolberg

12/01/2021, 9:27 PM

It does

Akiva Wolberg

12/01/2021, 9:29 PM

(I'm simultaneously trying to install R 4.0+ and the relevant packages in WSL, in case that's an easier solution, and having a terrible time for unrelated reasons. But morally, I feel like what I'm doing in prefect should work.)

Kevin Kho

12/01/2021, 9:30 PM

Let me open a Windows machine and get back to you in a bit

🙏🏻 1

Kevin Kho

12/01/2021, 9:44 PM

I got the same thing as you. Will work on it a bit

Kevin Kho

12/01/2021, 10:08 PM

Ok…I am pretty stuck. David here is a magician 😅. I get that pop up for the tempfile too and I think it blocks the operation. Will ask the team

Kevin Kho

12/01/2021, 10:20 PM

We do not support powershell: https://github.com/PrefectHQ/prefect/issues/3375

Akiva Wolberg

12/01/2021, 10:21 PM

Aha! That certainly answers that part. For cmd, should I submit a bug report? I really appreciate your time on this!

Kevin Kho

12/01/2021, 10:32 PM

The suggestion is to try running the Rscript yourself this way:

Copy code

import subprocess
from typing import Sequence, Union

@task
def run_process(command: Union[str, Sequence[str]]):
    if isinstance(command, str):
        command = command.split()
	return subprocess.run(command, check=True)

😀 1

👍🏻 1

🎉 1

Kevin Kho

12/01/2021, 10:33 PM

It’s a bit more work to get the logs. I think you can for

cmd

, but it’s honestly. a low priority ticket because there are so many gotchas with Windows and the recommendation is to use bash

Akiva Wolberg

12/01/2021, 10:35 PM

I think I figured out why it hangs---it's not actually hanging, it's just running an instance of command. When you get to the hang, type 'help' for instance. Which makes me think that the problem is actually that shelltask is starting in bash, loading cmd, and then stopping (maybe because you can't chain together bash commands once you're in cmd). example output:

Copy code

PS C:\Users\lwolberg\Desktop\workingdata> py r_shelltask.py
[2021-12-01 17:36:56-0500] INFO - prefect.FlowRunner | Beginning Flow run for 'abc'
[2021-12-01 17:36:56-0500] INFO - prefect.TaskRunner | Task 'ShellTask': Starting task run...
[2021-12-01 17:36:56-0500] INFO - prefect.ShellTask | Microsoft Windows [Version 10.0.19043.1165]
[2021-12-01 17:36:56-0500] INFO - prefect.ShellTask | (c) Microsoft Corporation. All rights reserved.
[2021-12-01 17:36:56-0500] INFO - prefect.ShellTask |
help
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | C:\Users\lwolberg\Desktop\workingdata>For more information on a specific command, type HELP command-name
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | ASSOC          Displays or modifies file extension associations.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | ATTRIB         Displays or changes file attributes.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | BREAK          Sets or clears extended CTRL+C checking.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | BCDEDIT        Sets properties in boot database to control boot loading.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | CACLS          Displays or modifies access control lists (ACLs) of files.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | CALL           Calls one batch program from another.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | CD             Displays the name of or changes the current directory.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | CHCP           Displays or sets the active code page number.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | CHDIR          Displays the name of or changes the current directory.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | CHKDSK         Checks a disk and displays a status report.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | CHKNTFS        Displays or modifies the checking of disk at boot time.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | CLS            Clears the screen.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | CMD            Starts a new instance of the Windows command interpreter.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | COLOR          Sets the default console foreground and background colors.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | COMP           Compares the contents of two files or sets of files.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | COMPACT        Displays or alters the compression of files on NTFS partitions.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | CONVERT        Converts FAT volumes to NTFS.  You cannot convert the
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |                current drive.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | COPY           Copies one or more files to another location.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | DATE           Displays or sets the date.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | DEL            Deletes one or more files.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | DIR            Displays a list of files and subdirectories in a directory.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | DISKPART       Displays or configures Disk Partition properties.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | DOSKEY         Edits command lines, recalls Windows commands, and
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |                creates macros.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | DRIVERQUERY    Displays current device driver status and properties.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | ECHO           Displays messages, or turns command echoing on or off.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | ENDLOCAL       Ends localization of environment changes in a batch file.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | ERASE          Deletes one or more files.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | EXIT           Quits the CMD.EXE program (command interpreter).
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | FC             Compares two files or sets of files, and displays the
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |                differences between them.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | FIND           Searches for a text string in a file or files.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | FINDSTR        Searches for strings in files.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | FOR            Runs a specified command for each file in a set of files.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | FORMAT         Formats a disk for use with Windows.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | FSUTIL         Displays or configures the file system properties.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | FTYPE          Displays or modifies file types used in file extension
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |                associations.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | GOTO           Directs the Windows command interpreter to a labeled line in
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |                a batch program.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | GPRESULT       Displays Group Policy information for machine or user.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | GRAFTABL       Enables Windows to display an extended character set in
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |                graphics mode.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | HELP           Provides Help information for Windows commands.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | ICACLS         Display, modify, backup, or restore ACLs for files and
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |                directories.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | IF             Performs conditional processing in batch programs.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | LABEL          Creates, changes, or deletes the volume label of a disk.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | MD             Creates a directory.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | MKDIR          Creates a directory.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | MKLINK         Creates Symbolic Links and Hard Links
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | MODE           Configures a system device.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | MORE           Displays output one screen at a time.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | MOVE           Moves one or more files from one directory to another
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |                directory.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | OPENFILES      Displays files opened by remote users for a file share.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | PATH           Displays or sets a search path for executable files.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | PAUSE          Suspends processing of a batch file and displays a message.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | POPD           Restores the previous value of the current directory saved by
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |                PUSHD.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | PRINT          Prints a text file.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | PROMPT         Changes the Windows command prompt.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | PUSHD          Saves the current directory then changes it.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | RD             Removes a directory.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | RECOVER        Recovers readable information from a bad or defective disk.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | REM            Records comments (remarks) in batch files or CONFIG.SYS.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | REN            Renames a file or files.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | RENAME         Renames a file or files.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | REPLACE        Replaces files.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | RMDIR          Removes a directory.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | ROBOCOPY       Advanced utility to copy files and directory trees
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | SET            Displays, sets, or removes Windows environment variables.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | SETLOCAL       Begins localization of environment changes in a batch file.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | SC             Displays or configures services (background processes).
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | SCHTASKS       Schedules commands and programs to run on a computer.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | SHIFT          Shifts the position of replaceable parameters in batch files.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | SHUTDOWN       Allows proper local or remote shutdown of machine.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | SORT           Sorts input.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | START          Starts a separate window to run a specified program or command.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | SUBST          Associates a path with a drive letter.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | SYSTEMINFO     Displays machine specific properties and configuration.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | TASKLIST       Displays all currently running tasks including services.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | TASKKILL       Kill or stop a running process or application.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | TIME           Displays or sets the system time.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | TITLE          Sets the window title for a CMD.EXE session.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | TREE           Graphically displays the directory structure of a drive or
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |                path.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | TYPE           Displays the contents of a text file.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | VER            Displays the Windows version.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | VERIFY         Tells Windows whether to verify that your files are written
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |                correctly to a disk.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | VOL            Displays a disk volume label and serial number.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | XCOPY          Copies files and directory trees.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | WMIC           Displays WMI information inside interactive command shell.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask | For more information on tools see the command-line reference in the online help.
[2021-12-01 17:37:04-0500] INFO - prefect.ShellTask |

7 Views

Open in Slack

Previous Next