<@ULVA73B9P> I got the below error when running a ...
# ask-marvin
c
@Marvin I got the below error when running a flow. Can you help me fix it?
FileNotFoundError: [Errno 2] No such file or directory: 'powershell.exe'
m
thought for 243.6 seconds
That error means your flow tried to launch PowerShell as the executable name
powershell.exe
, but the runtime can’t find it on the PATH (or it isn’t installed). This often happens when: - You’re running on Linux/macOS (no Windows PowerShell), or - You explicitly set
shell="powershell.exe"
(instead of
powershell
or
pwsh
), or - You’re in a minimal Windows environment/container without PowerShell on PATH. Quick ways to fix it - If you’re on Linux/macOS and just need a shell: - Use bash/sh explicitly. - Example with prefect-shell:
Copy code
pip install prefect-shell
Copy code
from prefect import flow
  from prefect_shell import shell_run_command

  @flow
  async def run():
      out = await shell_run_command("echo hello", shell="bash")
      print(out)
- If you need PowerShell cross‑platform: - Install PowerShell Core (pwsh) and use
shell="pwsh"
. - Example:
Copy code
from prefect import flow
  from prefect_shell import shell_run_command

  @flow
  async def run():
      out = await shell_run_command("Get-Process", shell="pwsh")
      print(out)
- If you’re on Windows and want Windows PowerShell: - Use
shell="powershell"
(do not use
powershell.exe
). - Ensure
C:\Windows\System32\WindowsPowerShell\v1.0\
is on PATH. - Example:
Copy code
from prefect import flow
  from prefect_shell import shell_run_command

  @flow
  async def run():
      out = await shell_run_command("Get-Process", shell="powershell")
      print(out)
Notes - Avoid
.ps1
scripts on non‑Windows unless you have
pwsh
installed. - If you’re calling
subprocess
yourself inside a task, swap
["powershell.exe", ...]
for
["powershell", ...]
on Windows or
["pwsh", ...]
cross‑platform. Can you share: - Where this is running (OS / container image)? - Whether you’re using
prefect_shell
(ShellOperation/shell_run_command) or
subprocess
? - The code snippet where the shell is invoked? With that, I can give a precise fix for your setup.
c
@Marvin How do I execute a powershell script in Prefect Python and have the flow return the output of that Powershell script?
m
thought for 197.8 seconds
Great question! In Prefect 3.x the easiest, supported way to run PowerShell and capture its output is the prefect-shell integration. Quick start 1) Install the integration
Copy code
pip install "prefect[shell]"
# or
pip install prefect-shell
2) Run a PowerShell command and return its output from your flow
Copy code
from prefect import flow
from prefect_shell import shell_run_command

@flow
def run_powershell_inline():
    # return_all=True -> capture full stdout as a list of lines
    out = shell_run_command(
        command='Write-Output "Hello from PowerShell"; Get-Date',
        shell="powershell",      # or "pwsh" for PowerShell 7+
        return_all=True
    )
    return out

result = run_powershell_inline()
print(result)
Run an existing .ps1 script (with arguments)
Copy code
from prefect import flow
from prefect_shell import shell_run_command

@flow
def run_ps_script(script_path: str, name: str):
    # Use the call operator & to execute the script file
    cmd = f'& "{script_path}" -Name "{name}"'
    out = shell_run_command(
        command=cmd,
        shell="powershell",      # or "pwsh"
        return_all=True
    )
    return out

result = run_ps_script("C:\\scripts\\hello.ps1", "Marvin")
print(result)
Tips - Choose your shell explicitly: -
shell="powershell"
for Windows PowerShell (5.1) -
shell="pwsh"
for PowerShell 7+ (cross‑platform) - Return format: -
return_all=False
(default) returns only the last line as a string -
return_all=True
returns all stdout lines as a list[str] - Non-zero exit codes raise a RuntimeError automatically (Prefect appends
Exit $LastExitCode
to propagate PS exit codes). Wrap in try/except if you want to handle failures:
Copy code
from prefect import flow, get_run_logger
from prefect_shell import shell_run_command

@flow
def robust_run():
    log = get_run_logger()
    try:
        return shell_run_command(command='throw "oops"', shell="powershell")
    except RuntimeError as e:
        log.error(f"PowerShell failed: {e}")
        return None
- Working directory and environment:
Copy code
shell_run_command(
    command='Get-Item . | Select-Object -ExpandProperty FullName',
    helper_command='cd C:\\Projects\\MyApp',
    env={"APP_ENV": "prod"},
    shell="powershell",
    return_all=True,
)
- If you hit Execution Policy errors on Windows, set a process-scoped policy first:
Copy code
shell_run_command(
    command='Write-Output "Policy ok"; & "C:\\scripts\\doit.ps1"',
    helper_command='Set-ExecutionPolicy -Scope Process Bypass -Force',
    shell="powershell",
    return_all=True,
)
Advanced: ShellOperation block For multi-step or long-running scripts with streaming logs and reusability:
Copy code
from prefect import flow
from prefect_shell import ShellOperation

@flow
def multi_step():
    op = ShellOperation(
        commands=[
            'Write-Host "Step 1"',
            'Write-Host "Step 2"',
        ],
        shell="powershell",
        stream_output=True
    )
    return op.run()  # returns list[str]
Docs and references - prefect-shell docs - SDK reference - Source on GitHub - PyPI: prefect-shell Would you like me to tailor an example to your exact setup? If you can share: - Windows vs. Linux/macOS - PowerShell version (powershell.exe vs pwsh) - Whether you’re calling a .ps1 file and what arguments it needs …I can provide a drop-in flow snippet.
c
@Marvin My output usually looks like this below instead of the expected string output. How do I extract the actual output of my script from this result?
<coroutine object run_flow_async at 0x0000020BA3C595A0>
m
thought for 105.5 seconds
That string means you’re returning an un-awaited coroutine. In other words, something async in your flow (the flow itself or the PowerShell call) wasn’t awaited before being returned. Fix it in one of these ways: 1) Make the flow async and await the command
Copy code
from prefect import flow
from prefect_shell import shell_run_command

@flow
async def run_ps():
    out = await shell_run_command(
        command='Write-Output "Hello"',
        shell="powershell",
        return_all=True
    )
    return out

result = run_ps()
print(result)
2) Keep the flow sync by using the synchronous ShellOperation API
Copy code
from prefect import flow
from prefect_shell import ShellOperation

@flow
def run_ps():
    out = ShellOperation(
        commands=['Write-Output "Hello"'],
        shell="powershell"
    ).run()
    return out

result = run_ps()
print(result)
3) If you used
.submit()
(i.e., ran it as a task), get the value with
.result()
Copy code
from prefect import flow
from prefect_shell import shell_run_command

@flow
def run_ps():
    fut = shell_run_command.submit(
        command='Write-Output "Hello"',
        shell="powershell",
        return_all=True
    )
    return fut.result()

print(run_ps())
If you share a small snippet of your flow, I can point out exactly where the missing
await
(or
.result()
) is. Refs: - prefect-shell docs - prefect-shell API
c
@Marvin for additional context, I am running the below Python script and expecting out to be a string containing a bearer token.
Copy code
async def getToken():
    # Path to your PowerShell script
    powershell_script_path = "Get-Token-Intune.ps1"
    #command = ["powershell.exe", "-ExecutionPolicy", "Bypass", "-File", powershell_script_path]

    #token = subprocess.run(command, capture_output=True, text=True)

    out = await shell_run_command(powershell_script_path, shell="powershell", return_all=True)
    return out
m
thought for 43.9 seconds
You’re very close—two key things to fix: 1) You need to await the function that calls shell_run_command (otherwise you’ll see a coroutine object), and 2) return_all=True gives you a list of lines; if you want a single string token, either set return_all=False or pick/join the lines. Also, when executing a .ps1 file via PowerShell, invoke it with the call operator
&
so it actually runs the script. Here are two working patterns. A) Keep it async and return a single string token
Copy code
import asyncio
from prefect_shell import shell_run_command

async def getToken():
    powershell_script_path = "C:\\path\\to\\Get-Token-Intune.ps1"
    token = await shell_run_command(
        command=f'& "{powershell_script_path}"',   # invoke the script
        shell="powershell",                        # or "pwsh"
        return_all=False,                          # last line as a single string
        # optional if you hit ExecutionPolicy issues:
        helper_command='Set-ExecutionPolicy -Scope Process Bypass -Force'
    )
    return token.strip()

# If calling from plain Python (not inside an async Prefect flow)
if __name__ == "__main__":
    print(asyncio.run(getToken()))
If you call getToken() from an async Prefect flow, make sure you await it:
Copy code
from prefect import flow

@flow
async def my_flow():
    token = await getToken()
    return token
B) Make it synchronous using ShellOperation
Copy code
from prefect_shell import ShellOperation

def getToken_sync():
    powershell_script_path = "C:\\path\\to\\Get-Token-Intune.ps1"
    lines = ShellOperation(
        commands=[f'& "{powershell_script_path}"'],
        shell="powershell",
        stream_output=False,
    ).run()
    return lines[-1].strip()  # last line as your token
PowerShell output tips - Make sure your .ps1 emits just the token on stdout. Prefer Write-Output $token (not Write-Host) for captured output. - If the script writes multiple lines, return_all=False returns only the last line; or use return_all=True and pick the line you need:
token = (await ...)[-1].strip()
. If you can share how you’re calling getToken() (from a flow or a standalone script) and what the script prints, I can tailor this exactly. Refs: - prefect-shell docs - prefect-shell API
c
@Marvin I updated my script accordingly to below. However, the flow is returning a blank string instead of the token.
Copy code
#Get Raw Data from API
@flow (retries = 0, retry_delay_seconds = 20)
async def getToken():
    # Path to your PowerShell script
    powershell_script_path = "Get-Token-Intune.ps1"
    #command = ["powershell.exe", "-ExecutionPolicy", "Bypass", "-File", powershell_script_path]

    #token = subprocess.run(command, capture_output=True, text=True)

    out = await shell_run_command(powershell_script_path, shell="powershell", return_all=False)
    return out.strip()
m
thought for 52.8 seconds
Thanks for the snippet — a blank string usually means your PowerShell script isn’t writing anything to stdout that Prefect can capture. A few fixes/checks to apply: 1) Invoke the script correctly Pass a command that actually runs the .ps1 using the call operator and quotes. Right now you’re passing just the path, which ends up as the content of a temp script. Use this:
Copy code
out = await shell_run_command(
    command=f'& "{powershell_script_path}"',
    shell="powershell",
    return_all=False
)
return out.strip()
2) Ensure your .ps1 emits the token to stdout Prefect captures stdout. In PowerShell, prefer
Write-Output $token
or just
$token
as the last statement. Avoid: -
Write-Host
(goes to the host/information stream, not stdout) - `Write-Verbose`/`Write-Information` (not stdout) -
Out-Null
(swallows output) -
Exit 0
(exits with success but no output) Minimal example inside your .ps1:
Copy code
# ... your logic to get $token ...
Write-Output $token
3) Inspect what the script prints Temporarily capture all output lines so you can see what’s coming back:
Copy code
lines = await shell_run_command(
    command=f'& "{powershell_script_path}"',
    shell="powershell",
    return_all=True
)
print(lines)  # for debugging
token = lines[-1].strip() if lines else ""
4) Common Windows gotchas - If you hit execution policy issues, add a helper command:
Copy code
out = await shell_run_command(
    command=f'& "{powershell_script_path}"',
    helper_command='Set-ExecutionPolicy -Scope Process Bypass -Force',
    shell="powershell",
    return_all=False
)
- Use an absolute path and quote it:
C:\\path\\to\\Get-Token-Intune.ps1
- If your script relies on relative paths, set
cwd
or use
helper_command='cd C:\\path\\to'
If you still get an empty string, can you share the last few lines of your PowerShell script (redacting secrets)? In particular, how the token is produced and printed. I can adjust the command or the script to ensure the token is captured. Refs: prefect-shell docs | API reference
c
@Marvin How do I run a powershell script called "Get-Token-Intune.ps1" and save the string that it returns as a variable?
m
thought for 28.3 seconds
Shortest path with prefect-shell: - Make sure your .ps1 writes the token to stdout (use Write-Output $token or just $token as the last statement; avoid Write-Host). - Invoke the script with PowerShell’s call operator
&
and capture the last line. Async flow using shell_run_command (returns a single string):
Copy code
from prefect import flow
from prefect_shell import shell_run_command

@flow
async def fetch_token():
    token = await shell_run_command(
        command='& "C:\\path\\to\\Get-Token-Intune.ps1"',  # or '& ".\\Get-Token-Intune.ps1"'
        shell="powershell",     # or "pwsh" for PowerShell 7+
        return_all=False        # last line as a single string
        # helper_command='Set-ExecutionPolicy -Scope Process Bypass -Force',  # if needed
        # cwd="C:\\path\\to\\scripts",  # if your script uses relative paths
    )
    return token.strip()

result = fetch_token()
print(result)
Synchronous alternative with ShellOperation:
Copy code
from prefect import flow
from prefect_shell import ShellOperation

@flow
def fetch_token_sync():
    lines = ShellOperation(
        commands=['& "C:\\path\\to\\Get-Token-Intune.ps1"'],
        shell="powershell",
        stream_output=False
    ).run()
    token = lines[-1].strip() if lines else ""
    return token

print(fetch_token_sync())
If you still get an empty string: - Double-check the script prints the token on stdout (Write-Output $token; not Write-Host). - Temporarily debug with return_all=True to see everything the script emits:
Copy code
lines = await shell_run_command(command='& ".\\Get-Token-Intune.ps1"', shell="powershell", return_all=True)
print(lines)
Docs: - prefect-shell docs - API reference
c
@Marvin I deployed my script and got the following error. Do you know how to solve it? I am pulling my imports from a Docker image.
FileNotFoundError: [Errno 2] No such file or directory: 'powershell'
m
thought for 242.6 seconds
Short answer: something in your flow is trying to invoke PowerShell, but your runtime is a Linux container that doesn’t have it (and Linux doesn’t provide a
powershell
binary). This commonly happens when: - Using
prefect-shell
with
shell="powershell"
(or PowerShell-only commands) - A library (e.g., Azure Identity) tries to fall back to PowerShell on Windows, but your flow actually runs in Linux Ways to fix it (pick one): 1) If you’re using prefect-shell, run bash instead of PowerShell in Linux containers - Update your flow to use bash-compatible commands:
Copy code
from prefect_shell import shell_run_command

# Recommended when running in a Linux image
result = shell_run_command(command="echo hello", shell="bash")
- If you need cross-platform behavior, choose shell by platform:
Copy code
import sys
from prefect_shell import shell_run_command

shell = "powershell" if sys.platform == "win32" else "bash"
result = shell_run_command(command="echo hello", shell=shell)
Reference: prefect-shell integration 2) If you truly need PowerShell, run on Windows or build an image that includes PowerShell - Prefect’s official images are Linux and don’t include PowerShell. If you must use PS in a Linux container, install it and invoke
pwsh
(note: on Linux the executable is
pwsh
, not
powershell
, and exit code semantics differ). Microsoft docs: Install PowerShell on Linux 3) If you’re using Azure Identity, disable its PowerShell credential - In code:
Copy code
from azure.identity import DefaultAzureCredential
cred = DefaultAzureCredential(exclude_powershell_credential=True)
- Or set env var:
Copy code
AZURE_IDENTITY_DISABLE_POWERSHELL_CREDENTIAL=true
Docs: Azure Identity credential chain A couple of clarifying questions so I can tailor the fix: - Are you using
prefect-shell
anywhere (e.g.,
shell_run_command
or
ShellOperation
)? - Are you using Azure Identity or any library that might spawn PowerShell? - What image are you running your flow in? If you can share your Dockerfile or the image tag and a short snippet of the code that runs a shell command, I can give you an exact patch. General note: when you deploy flows to run from a Docker image in Prefect 3.x, they’ll typically run in Linux containers. Windows-specific commands (e.g.,
powershell
,
dir
) will fail unless you target Windows infrastructure or adjust your commands. Deployment docs: Deploy flows with Prefect
c
@Marvin I received the following error. Can you explain it? `Finished in state Failed('Task run encountered an exception RuntimeError: Command failed with exit code 2\n/tmp/prefect tc6cd5qz line 1: syntax error near unexpected token `&\'\n/tmp/prefect-tc6cd5qz: line 1:
& ".\\Get-Token-Intune.ps1"\'\n')
m
🔄 Thinking... this may take a while
Thanks for sharing the full error — here’s what it means and how to fix it. What happened - Your task ran a shell command that starts with
& ".\Get-Token-Intune.ps1"
. - The
&
you see is just HTML-escaping of
&
— the real command was
& ".\Get-Token-Intune.ps1"
. -
&
and
.\
are PowerShell syntax. But your flow ran in a Linux environment using bash (Prefect writes your command to a temp file like
/tmp/prefect-xxxx
and executes it with the chosen shell). - Bash tried to parse the PowerShell command and failed with a syntax error. Exit code 2 is a bash “syntax misuse” error. Why this happens in Prefect - When using
prefect-shell
(e.g.,
shell_run_command
or
ShellOperation
), the default shell is: - PowerShell on Windows - bash on Linux/macOS - If you deploy to/run in a Linux container (Prefect’s default images are Linux), your PowerShell command gets executed by bash unless you explicitly run PowerShell. How to fix Pick one of these approaches: 1) Use bash-safe commands (recommended if you don’t need PowerShell) - Replace PowerShell syntax with bash equivalents. - Example:
Copy code
from prefect_shell import shell_run_command

result = shell_run_command(command="echo hello", shell="bash")
2) If you need to run a PowerShell script, run PowerShell explicitly (and ensure it’s installed) - In Linux, the executable is typically
pwsh
(PowerShell 7), not
powershell
. - Update your command and/or shell:
Copy code
from prefect_shell import shell_run_command

# Option A: keep bash as the shell, call pwsh directly
result = shell_run_command(
    command='pwsh -NoProfile -File ./Get-Token-Intune.ps1',
    shell='bash'
)

# Option B: use pwsh as the shell
result = shell_run_command(
    command='./Get-Token-Intune.ps1',
    shell='pwsh'
)
- Note the Linux-style path
./Get-Token-Intune.ps1
(not
.\
). - Make sure your Docker image installs PowerShell: - Install guide: Install PowerShell on Linux 3) If this script is only for getting tokens (e.g., Microsoft Intune/Azure), consider using the Python SDK directly to avoid spawning a shell - For Azure authentication,
azure-identity
with
DefaultAzureCredential
often replaces custom PS scripts:
Copy code
from azure.identity import DefaultAzureCredential
cred = DefaultAzureCredential()
Docs: Azure Identity credential chain Helpful reference for shell tasks: prefect-shell integration If you can share: - The snippet where you call
prefect-shell
, and - The base image/Dockerfile you’re using, I can suggest the exact change (including whether to set
shell="pwsh"
or call
pwsh -File ...
directly).
c
@Marvin Now I am getting the below error. Any ideas? The file is in the same directory that I am executing the code from.
Copy code
RuntimeError: Command failed with exit code 127:
/tmp/prefect-kq75xre5: line 1: .\Get-Token-Intune.ps1: command not found
m
thought for 49.4 seconds
Exit code 127 means “command not found.” In your case, bash is trying to run:
Copy code
.\Get-Token-Intune.ps1
That path is PowerShell/Windows syntax (
.\
). In a Linux container with bash: -
.\script.ps1
is not a valid command (hence 127) - Bash can’t execute a PowerShell script unless you invoke PowerShell explicitly - Even if the file is “in the same directory” on your machine, it must also exist in the container’s working directory at runtime How to fix it Pick one of these patterns: 1) Run the script with PowerShell (Linux) and use Linux-style paths - Make sure PowerShell 7 is installed in the image (the executable is
pwsh
on Linux) - Use
./Get-Token-Intune.ps1
(not
.\...
) and either call
pwsh -File ...
or set the shell to
pwsh
Example A: keep bash as shell, call pwsh directly
Copy code
from prefect_shell import shell_run_command

result = shell_run_command(
    command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./Get-Token-Intune.ps1',
    shell='bash',                # your container’s default
    cwd='/app'                   # set if needed
)
Example B: use pwsh as the shell
Copy code
from prefect_shell import shell_run_command

result = shell_run_command(
    command='./Get-Token-Intune.ps1',
    shell='pwsh',
    cwd='/app'                   # set if needed
)
PowerShell install guide (Linux): Install PowerShell on Linux prefect-shell reference: prefect-shell integration 2) Ensure the script is actually present in the container and in the working directory - If you’re using a Docker image, copy the script into it and set the working directory, e.g.:
Copy code
WORKDIR /app
COPY Get-Token-Intune.ps1 /app/
- Or set
cwd
in
shell_run_command
to the directory where the script lives - Quick sanity check inside the task:
Copy code
shell_run_command(command='pwd && ls -la', shell='bash')
3) If you don’t need PowerShell, convert to a bash command or Python code - For Azure tokens, the Python SDK can often replace PS scripts:
Copy code
from azure.identity import DefaultAzureCredential
cred = DefaultAzureCredential()
Docs: Azure Identity credential chain If you share: - How you’re calling
shell_run_command
(snippet), - Your Dockerfile or image tag, I can pinpoint whether you need
shell='pwsh'
,
pwsh -File ...
,
cwd=...
, or a Dockerfile change.
c
@Marvin Can you explain this error I got while running a .ps1 file in Prefect?
Copy code
RuntimeError: Command failed with exit code 2:
./Get-Token-Intune.ps1: line 4: =: command not found
./Get-Token-Intune.ps1: line 5: =: command not found
./Get-Token-Intune.ps1: line 6: =: command not found
./Get-Token-Intune.ps1: line 9: =: command not found
./Get-Token-Intune.ps1: line 10: =: command not found
./Get-Token-Intune.ps1: line 11: =: command not found
./Get-Token-Intune.ps1: line 12: resource: command not found
./Get-Token-Intune.ps1: line 13: client_id: command not found
./Get-Token-Intune.ps1: line 14: client_secret: command not found
./Get-Token-Intune.ps1: line 15: grant_type: command not found
./Get-Token-Intune.ps1: line 16: scope: command not found
./Get-Token-Intune.ps1: line 17: syntax error near unexpected token `}'
./Get-Token-Intune.ps1: line 17: `}'
m
thought for 57.2 seconds
Those errors mean your PowerShell script is being interpreted by bash, not PowerShell. In a Linux runtime (e.g., a Docker container), if you execute
./Get-Token-Intune.ps1
without telling the system to use PowerShell (or without a PowerShell shebang), bash will try to parse the file. Lines like
$var = ...
or hashtables become “= …” and “resource:” to bash, which produces “command not found” and a final syntax error. Exit code 2 is bash’s “syntax misuse.” How to fix it Pick one of these: 1) Run the script with PowerShell (Linux) and use Linux-style paths - Ensure PowerShell 7 is installed in your image (the binary is
pwsh
on Linux) - Either set the Prefect shell to
pwsh
, or call
pwsh -File ...
from bash Example A: use pwsh as the shell
Copy code
from prefect_shell import shell_run_command

shell_run_command(
    command='./Get-Token-Intune.ps1',   # PowerShell will interpret this
    shell='pwsh',
    cwd='/app'                          # set if needed
)
Example B: keep bash as shell, invoke pwsh explicitly
Copy code
from prefect_shell import shell_run_command

shell_run_command(
    command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./Get-Token-Intune.ps1',
    shell='bash',
    cwd='/app'
)
2) Add a PowerShell shebang to the script so the OS picks the right interpreter - At the top of Get-Token-Intune.ps1:
Copy code
#!/usr/bin/env pwsh
- Make it executable:
Copy code
chmod +x Get-Token-Intune.ps1
- Then
./Get-Token-Intune.ps1
will run under PowerShell even when launched from bash. 3) If you don’t need PowerShell, convert the logic to bash or Python (e.g., Azure auth via
azure-identity
) to avoid spawning PowerShell. Also check: - Use
./Get-Token-Intune.ps1
(Linux style), not
.\...
- Ensure the script is actually in the container’s working directory (set
cwd
or verify with
pwd && ls -la
) - Make sure the file has LF line endings; CRLF can cause odd shell parsing issues. If needed:
dos2unix Get-Token-Intune.ps1
References: - prefect-shell integration: prefect-shell - Install PowerShell on Linux: Microsoft docs If you share your
shell_run_command
call and Dockerfile/image, I can recommend the exact change (pwsh vs pwsh -File, cwd, permissions).
c
My shell run command is:
await shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./Get-Token-Intune.ps1', shell="bash", return_all=False)
@Marvin My shell run command is:
await shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./Get-Token-Intune.ps1', shell="bash", return_all=False)
and my docker image is:
Copy code
docker image is: FROM prefecthq/prefect:3.2.14-python3.9
COPY . /opt/prefect/Scripts/
WORKDIR /opt/prefect/Scripts/
RUN pip install --no-cache-dir prefect-snowflake GitPython PyGithub "prefect[shell]" "snowflake-connector-python[pandas]==4.0.0" "pandas==2.3.3" "numpy>=2.0,<3" "pyarrow==21.0.0" "prefect[shell]" adlfs s3fs requests pyodbc prefect_azure && apt-get update && apt-get install -y unixodbc -y curl ca-certificates && curl -sSL -o /usr/local/bin/kubectl <https://dl.k8s.io/release/v1.30.0/bin/linux/amd64/kubectl> && chmod +x /usr/local/bin/kubectl && apt-get update && apt-get install -y wget software-properties-common && wget -q "<https://packages.microsoft.com/config/debian/$(lsb_release> -rs)/packages-microsoft-prod.deb" -O packages-microsoft-prod.deb && dpkg -i packages-microsoft-prod.deb && rm packages-microsoft-prod.deb && apt-get update
m
thought for 61.7 seconds
Thanks for the details — your Prefect call looks fine. The issue is almost certainly your image doesn’t actually have PowerShell installed. What I’m seeing: - You’re running:
pwsh -NoProfile -ExecutionPolicy Bypass -File ./Get-Token-Intune.ps1
with
shell="bash"
. That’s good — bash will spawn pwsh to run the script. - In your Dockerfile you add Microsoft’s repo (via packages-microsoft-prod.deb) but you never install PowerShell. There’s no
apt-get install -y powershell
, so
pwsh
won’t exist in the container. - Minor: your snippet shows HTML-escaped characters (
&&
,
>
,
<
). Make sure your real Dockerfile uses
&&
,
>=
,
<
etc. What to change in your Dockerfile Add an explicit install of PowerShell (pwsh). Also, avoid relying on
lsb_release
and use the codename from /etc/os-release to pick the right Microsoft repo:
Copy code
FROM prefecthq/prefect:3.2.14-python3.9

# Copy your code
COPY . /opt/prefect/Scripts/
WORKDIR /opt/prefect/Scripts/

# Python deps (dedupe prefect[shell] if you had it twice; optionally use prefect-shell explicitly)
RUN pip install --no-cache-dir \
    prefect-snowflake \
    GitPython \
    PyGithub \
    "prefect[shell]" \
    "snowflake-connector-python[pandas]==4.0.0" \
    "pandas==2.3.3" \
    "numpy>=2.0,<3" \
    "pyarrow==21.0.0" \
    adlfs s3fs requests pyodbc prefect_azure

# OS deps + PowerShell
RUN apt-get update && apt-get install -y --no-install-recommends \
      ca-certificates curl wget gnupg software-properties-common \
      unixodbc \
    && . /etc/os-release \
    && wget -q "<https://packages.microsoft.com/config/debian/${VERSION_CODENAME}/packages-microsoft-prod.deb>" -O packages-microsoft-prod.deb \
    && dpkg -i packages-microsoft-prod.deb \
    && rm packages-microsoft-prod.deb \
    && apt-get update \
    && apt-get install -y --no-install-recommends powershell \
    && curl -sSL -o /usr/local/bin/kubectl <https://dl.k8s.io/release/v1.30.0/bin/linux/amd64/kubectl> \
    && chmod +x /usr/local/bin/kubectl \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
Sanity checks (run once in your flow or container) - Verify PowerShell is present and in PATH:
Copy code
await shell_run_command(command='which pwsh && pwsh -v', shell='bash')
- Verify the script exists where you expect:
Copy code
await shell_run_command(command='pwd && ls -la', shell='bash', return_all=True)
- Then run your script (what you already have is fine):
Copy code
await shell_run_command(
    command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./Get-Token-Intune.ps1',
    shell='bash',
    return_all=False,
    cwd='/opt/prefect/Scripts'   # optional, but removes any doubt about CWD
)
A few additional tips - Use Linux paths (
./Get-Token-Intune.ps1
, not
.\...
). - Ensure the script has LF line endings (not CRLF). If needed during build: - Install dos2unix:
apt-get install -y dos2unix
then
dos2unix /opt/prefect/Scripts/Get-Token-Intune.ps1
- You don’t need the script to be executable when using
pwsh -File
, but it must be readable and present in the working directory. - If you prefer, you can also run with
shell="pwsh"
and
command='./Get-Token-Intune.ps1'
once pwsh is installed. References: - prefect-shell: prefect-shell integration - Install PowerShell on Linux: Microsoft docs If this still errors after adding
apt-get install -y powershell
, share the new error text and I’ll help debug further.
c
@Marvin Where is the unexpected indentation in this code?
Copy code
import requests, json, pyodbc, pandas as pd, re
from prefect import flow, task
from azure.storage.blob import BlobClient, BlobServiceClient
from azure.identity import DefaultAzureCredential
from prefect_snowflake.credentials import SnowflakeCredentials
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.backends import default_backend
from prefect_shell import shell_run_command
#from prefect.filesystems import Azure
import snowflake.connector
import subprocess
import asyncio

#Get Raw Data from API
@flow (retries = 0, retry_delay_seconds = 20)
async def getData():
    # Path to your PowerShell script
    powershell_script_path = "DefenderGetAlerts.ps1"

    out = await shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1', shell="bash", return_all=False)
    return out.strip()

#Load JSON data into Snowflake
@flow (retries = 0, retry_delay_seconds = 20, log_prints=True)
def loadJson(data):
    print(data)
    blob_service_client = BlobServiceClient(
        account_url="<https://prefectscriptspy.blob.core.windows.net/>",
        credential='mec3I4AojCiv764tjydLxY8twl16IwIe+9iJVsWgQnG2V+aAsrEIhhcUozcMMRxVIsQKz43V82Un+ASttIsz3w==')
    
    container_name = "prefectscripts"
    blob_name = "rsa_key.p8" # e.g., "private_key.pem"

    container_client = blob_service_client.get_container_client(container_name)
    blob_client = container_client.get_blob_client(blob_name)

    try:
        download_stream = blob_client.download_blob()
        private_key_content = download_stream.readall()
        # If the private key is text-based (e.g., PEM), you might want to decode it
        # private_key_string = private_key_content.decode("utf-8")
        
        print("Private key retrieved successfully.")
        # Now, private_key_content holds the bytes of your private key
        # You can process it further, e.g., load it into a cryptographic library
    except Exception as e:
        print(f"Error retrieving private key: {e}")
    
    p_key = serialization.load_pem_private_key(
        private_key_content,
        password=None,  # Set to your password if the key is encrypted
        backend=default_backend()
        )

    USER = 'SVC_PF_USER'
    ACCOUNT = 'xz30609.central-us.azure'
    WAREHOUSE = 'COMPUTE_WH'
    DATABASE = 'UNITEDCENTER_PROD_RAW_DB'
    SCHEMA = 'MICROSOFT_SECURITY'

    conn = snowflake.connector.connect(
    user=USER,
    private_key = p_key,
    account=ACCOUNT,
    warehouse=WAREHOUSE,
    database=DATABASE,
    schema=SCHEMA)
    conn.timeout = 6000   
    conn.autocommit = True

    dataToLoad = json.loads(data)
    dataToLoad = json.dumps(dataToLoad)
    # Call SP and trap Error if raised
    cursor = conn.cursor()
        # Execute Transactions load
    cursor.execute('CALL load_defender_data(%s)', dataToLoad)
    print('inserted data')    

    #except pyodbc.Error as err:
    #    print('Error !!!!! %s' % err)
    #except:
     #   print('something else failed miserably')

    conn.close()
    print('closed db connection')

@flow (retries = 3, retry_delay_seconds = 20, log_prints=True)
def main_flow():
    jsonData = asyncio.run(getData())
    print(jsonData)
    loadJson(jsonData)

if __name__ == '__main__':
    jsonData = asyncio.run(getData())
    print(type(jsonData))
    loadJson(jsonData)
m
thought for 51.4 seconds
The only thing that jumps out is this extra-indented comment between your two cursor lines:
Copy code
cursor = conn.cursor()
    # Execute Transactions load   <-- extra indent here
cursor.execute('CALL load_defender_data(%s)', dataToLoad)
Dedent that comment so it aligns with the surrounding lines:
Copy code
cursor = conn.cursor()
# Execute Transactions load
cursor.execute('CALL load_defender_data(%s)', dataToLoad)
Also double-check there aren’t any stray tabs mixed with spaces around that spot. If you still get an IndentationError, please share the full traceback with the reported line number and I’ll pinpoint it exactly.
c
@Marvin Can you explain why my shell_run_command call is responding with a null value? My call is:
out = await shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1', shell="bash", return_all=False)
And my .ps1 file is
Copy code
# Returns Alerts created in the past 48 hours.

$token = ./Get-Token.ps1       #run the script Get-Token.ps1  - make sure you are running this script from the same folder of Get-Token.ps1

# Get Alert from the last 48 hours. Make sure you have alerts in that time frame.
$dateTime = (Get-Date).ToUniversalTime().AddHours(-120).ToString("o")

# The URL contains the type of query and the time filter we create above
# Read more about [other query options and filters](get-alerts.md).
$url = "<https://api.securitycenter.microsoft.com/api/alerts>?`$filter=alertCreationTime ge $dateTime"

# Set the WebRequest headers
$headers = @{
    'Content-Type' = 'application/json'
    Accept = 'application/json'
    Authorization = "Bearer $token"
}

# Send the webrequest and get the results.
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop

# Extract the alerts from the results.
$alerts =  ($response | ConvertFrom-Json).value | ConvertTo-Json

# Get string with the execution time. We concatenate that string to the output file to avoid overwrite the file
$dateTimeForFileName = Get-Date -Format o | foreach {$_ -replace ":", "."}

Write-Output $alerts
return $alerts
m
thought for 127.5 seconds
Short version: with return_all=False, shell_run_command only returns the last line of stdout. If your PowerShell script emits multi-line JSON, or emits nothing, you’ll either get just the last line (which can be “}”, “]”, or “null”) or an empty string. Why you’re seeing a “null” (or empty) value - shell_run_command behavior: - return_all=False → returns a single string: the last stdout line - return_all=True → returns a list of all lines - It never returns None; if there’s no stdout, it returns "" - Reference: prefect-shell - Your script builds JSON with ConvertTo-Json, which by default pretty-prints across multiple lines. The “last line” might be “}” (or “null” if the value is $null). - If the API returns no alerts, ($response | ConvertFrom-Json).value may be $null. ConvertTo-Json $null yields the string null, so your “last line” might be “null”. - If your script ever writes nothing (e.g., Write-Output $null, return $null, or the pipeline produces $null), Prefect will get "" as the result. How to make the output predictable Pick one of these: 1) Make your PowerShell output a single line of JSON - Use -Compress so the entire JSON is emitted on one line - Avoid returning $null; coerce to an empty array/object - Avoid extraneous output (Write-Host, progress, etc.) Example ps1:
Copy code
# Returns Alerts created in the past 48 hours.

$token = ./Get-Token.ps1

# last 48 hours
$dateTime = (Get-Date).ToUniversalTime().AddHours(-48).ToString("o")

$url = "<https://api.securitycenter.microsoft.com/api/alerts?`$filter=alertCreationTime> ge $dateTime"
$headers = @{
  'Content-Type' = 'application/json'
  Accept         = 'application/json'
  Authorization  = "Bearer $token"
}

$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop

$alerts = ($response.Content | ConvertFrom-Json).value
if ($null -eq $alerts) { $alerts = @() }  # ensure array instead of $null

$alertsJson = $alerts | ConvertTo-Json -Compress
Write-Output $alertsJson
# no 'return' needed; the last expression or Write-Output sends to stdout
Then your Python call can stay as-is:
Copy code
out = await shell_run_command(
  command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1',
  shell='bash',
  return_all=False
)
# out will be a single-line JSON string: [] or [{"...": "..."}]
2) Capture all lines from the script and join in Python - Keep your current script, but change return_all to True:
Copy code
lines = await shell_run_command(
  command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1',
  shell='bash',
  return_all=True
)
out = "\n".join(lines)  # reconstruct the full JSON
Additional tips - If Get-Token.ps1 prints logging/status lines, those can become the “last line” and displace your JSON. Ensure it only writes the token to stdout, or assign its result with:
$token = & ./Get-Token.ps1
and keep that script quiet except for the token. - If there are no alerts, you’ll get [] (after the $null → @() coercion above) instead of null. - You can sanity-check what Prefect sees with:
Copy code
await shell_run_command(command='pwsh -c "Write-Output hello"', shell='bash', return_all=True)
If you share what exact string you get in Python (e.g., printing repr(out)), I can confirm whether it’s an empty string, the literal "null", or a truncated last line, and tailor the fix.
c
@Marvin I made changes to my .ps1 file and now I am getting the result
['null', 'null']
Updated .ps1 script is:
Copy code
# Returns Alerts created in the past 48 hours.

$token = ./Get-Token.ps1       #run the script Get-Token.ps1  - make sure you are running this script from the same folder of Get-Token.ps1

# Get Alert from the last 48 hours. Make sure you have alerts in that time frame.
$dateTime = (Get-Date).ToUniversalTime().AddHours(-120).ToString("o")

# The URL contains the type of query and the time filter we create above
# Read more about [other query options and filters](get-alerts.md).
$url = "<https://api.securitycenter.microsoft.com/api/alerts>?`$filter=alertCreationTime ge $dateTime"

# Set the WebRequest headers
$headers = @{
    'Content-Type' = 'application/json'
    Accept = 'application/json'
    Authorization = "Bearer $token"
}

# Send the webrequest and get the results.
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop

# Extract the alerts from the results.
$alerts =  ($response | ConvertFrom-Json).value | ConvertTo-Json -Compress

# Get string with the execution time. We concatenate that string to the output file to avoid overwrite the file
$dateTimeForFileName = Get-Date -Format o | foreach {$_ -replace ":", "."}

Write-Output $alerts
return $alerts
m
thought for 52.9 seconds
You’re seeing two lines of "null" because of two separate issues: 1) Duplicate output from your script - In the PS snippet you shared earlier, you had both: - Write-Output $alerts - return $alerts - In PowerShell, both of these write to the success output stream. That produces two identical lines, hence ['null', 'null'] when return_all=True. Fix: emit the result once (prefer Write-Output, or just output the final expression, and remove return). 2) $alerts is actually $null - In your earlier code you were doing: - $response = Invoke-WebRequest ... - $alerts = ($response | ConvertFrom-Json).value | ConvertTo-Json - ConvertFrom-Json expects JSON text. Piping the entire Invoke-WebRequest object into it yields $null. You need to parse $response.Content, or better, use Invoke-RestMethod which already parses JSON. Recommended PowerShell fix (single-line JSON, no duplicate output):
Copy code
$ErrorActionPreference = 'Stop'

$token = & ./Get-Token.ps1  # run script and capture only its output

# last 48 hours
$dateTime = (Get-Date).ToUniversalTime().AddHours(-48).ToString("o")

$url = "<https://api.securitycenter.microsoft.com/api/alerts?`$filter=alertCreationTime> ge $dateTime"
$headers = @{
  'Content-Type' = 'application/json'
  Accept         = 'application/json'
  Authorization  = "Bearer $token"
}

# Prefer Invoke-RestMethod for JSON
$response = Invoke-RestMethod -Method Get -Uri $url -Headers $headers

$alerts = $response.value
if ($null -eq $alerts) { $alerts = @() }  # ensure array instead of $null

# Emit a single line so Prefect can capture it as one item
$alerts | ConvertTo-Json -Compress
Then in Python: - If you want just one string line:
Copy code
out = await shell_run_command(
  command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1',
  shell='bash',
  return_all=False
)
# out is a single-line JSON string; parse if needed:
# data = json.loads(out)
- If you keep return_all=True, you should now get ['[...]'] with one element; you can join if needed:
Copy code
lines = await shell_run_command(..., return_all=True)
out = "\n".join(lines)
Why this works: - No “return” duplicates the output - Using Invoke-RestMethod or $response.Content ensures you actually parse JSON - -Compress keeps JSON to a single line so return_all=False gives you the whole payload If you still see "null", print the raw response for a quick check:
Copy code
pwsh -NoProfile -Command "Invoke-WebRequest -Uri '<your url>' -Headers <headers> | Select-Object -ExpandProperty Content | Set-Content -Path debug.json"
or switch to Invoke-RestMethod and inspect $response directly.
c
@Marvin why is my .ps1 file returning null?
Copy code
# Returns Alerts created in the past 48 hours.

$token = ./Get-Token.ps1       #run the script Get-Token.ps1  - make sure you are running this script from the same folder of Get-Token.ps1

# Get Alert from the last 48 hours. Make sure you have alerts in that time frame.
$dateTime = (Get-Date).ToUniversalTime().AddHours(-120).ToString("o")

# The URL contains the type of query and the time filter we create above
# Read more about [other query options and filters](get-alerts.md).
$url = "<https://api.securitycenter.microsoft.com/api/alerts>?`$filter=alertCreationTime ge $dateTime"

# Set the WebRequest headers
$headers = @{
    'Content-Type' = 'application/json'
    Accept = 'application/json'
    Authorization = "Bearer $token"
}

# Send the webrequest and get the results.
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop

# Extract the alerts from the results.
$alerts =  ($response.Content | ConvertFrom-Json).value | ConvertTo-Json -Compress

# Get string with the execution time. We concatenate that string to the output file to avoid overwrite the file
$dateTimeForFileName = Get-Date -Format o | foreach {$_ -replace ":", "."}

Write-Output $alerts
m
thought for 68.5 seconds
Short answer: your script is emitting the JSON string "null" because the pipeline assigns $alerts to $null, then pipes it to ConvertTo-Json, which produces the literal text null. Prefect just captures what your script prints. Where it happens in your code
Copy code
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop
$alerts = ($response.Content | ConvertFrom-Json).value | ConvertTo-Json -Compress
Write-Output $alerts
- If
($response.Content | ConvertFrom-Json).value
is
$null
(e.g., no results, invalid token, or unexpected JSON shape), then piping
$null
to
ConvertTo-Json
yields the string
null
. - Prefect’s
shell_run_command(return_all=False)
returns the last stdout line; if that’s
null
, that’s what you’ll get. Common reasons
.value
is
$null
- The Defender API returned an error or a different JSON shape because the token is missing/invalid. A common cause is
Get-Token.ps1
printing via
Write-Host
(host stream), which won’t flow into
$token = ./Get-Token.ps1
. Use the call operator and ensure it writes to the success stream:
$token = (& ./Get-Token.ps1).Trim()
. - The URL string includes angle brackets (e.g.,
"<https://...>"
from Slack). Those
<
and
>
should not be in the URL. - The time filter returns no alerts and the API payload doesn’t include
value
(or it’s null). Some APIs return
{ value: [] }
, others may omit
value
. How to fix and make output predictable - Prefer Invoke-RestMethod to get parsed JSON automatically and coerce nulls to an empty array before converting to JSON:
Copy code
$ErrorActionPreference = 'Stop'

$token = (& ./Get-Token.ps1).Trim()
if ([string]::IsNullOrWhiteSpace($token)) { Write-Error "Empty token"; exit 1 }

# last 48 hours
$dateTime = (Get-Date).ToUniversalTime().AddHours(-48).ToString("o")

$url = "<https://api.securitycenter.microsoft.com/api/alerts?`$filter=alertCreationTime> ge $dateTime"
$headers = @{
  'Content-Type' = 'application/json'
  Accept         = 'application/json'
  Authorization  = "Bearer $token"
}

# Parse JSON automatically
$response = Invoke-RestMethod -Method Get -Uri $url -Headers $headers

$alerts = $response.value
if ($null -eq $alerts) { $alerts = @() }   # avoid null → "null"

# Emit a single line of JSON
$alerts | ConvertTo-Json -Compress
- If you keep Invoke-WebRequest, be sure to parse
.Content
(you are), then:
Copy code
$json = $response.Content | ConvertFrom-Json
$alerts = $json.value
if ($null -eq $alerts) { $alerts = @() }
$alerts | ConvertTo-Json -Compress
- Make sure the URL has no
<
or `>`:
Copy code
$url = "<https://api.securitycenter.microsoft.com/api/alerts?`$filter=alertCreationTime> ge $dateTime"
- Ensure
Get-Token.ps1
outputs only the token via success output (Write-Output or a bare expression), not Write-Host. In the caller, use the call operator:
Copy code
$token = (& ./Get-Token.ps1).Trim()
- For debugging, temporarily print what you’re actually getting:
Copy code
Write-Output "Token length: $($token.Length)"
Write-Output ($response | ConvertTo-Json -Compress)
Once
$alerts
is an array (even empty), Prefect will capture valid JSON instead of the string
null
.
c
@Marvin Where is the unexpected indentation in this code?
Copy code
import requests, json, pyodbc, pandas as pd, re
from prefect import flow, task
from azure.storage.blob import BlobClient, BlobServiceClient
from azure.identity import DefaultAzureCredential
from prefect_snowflake.credentials import SnowflakeCredentials
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.backends import default_backend
from prefect_shell import shell_run_command
import snowflake.connector
import subprocess
import asyncio

#Get Raw Data from API
@flow (retries = 0, retry_delay_seconds = 20)
async def getData():
    #Path to your PowerShell script
    powershell_script_path = "DefenderGetAlerts.ps1"

    out = await shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1', shell="bash", return_all=True)
	print(out)
    return str(out)

#Load JSON data into Snowflake
@flow (retries = 0, retry_delay_seconds = 20, log_prints=True)
def loadJson(data):
    print(data)
    blob_service_client = BlobServiceClient(
        account_url="<https://prefectscriptspy.blob.core.windows.net/>",
        credential='mec3I4AojCiv764tjydLxY8twl16IwIe+9iJVsWgQnG2V+aAsrEIhhcUozcMMRxVIsQKz43V82Un+ASttIsz3w==')
    
    container_name = "prefectscripts"
    blob_name = "rsa_key.p8" # e.g., "private_key.pem"

    container_client = blob_service_client.get_container_client(container_name)
    blob_client = container_client.get_blob_client(blob_name)

    try:
        download_stream = blob_client.download_blob()
        private_key_content = download_stream.readall()
        # If the private key is text-based (e.g., PEM), you might want to decode it
        # private_key_string = private_key_content.decode("utf-8")
        
        print("Private key retrieved successfully.")
        # Now, private_key_content holds the bytes of your private key
        # You can process it further, e.g., load it into a cryptographic library
    except Exception as e:
        print(f"Error retrieving private key: {e}")
    
    p_key = serialization.load_pem_private_key(
        private_key_content,
        password=None,  # Set to your password if the key is encrypted
        backend=default_backend()
        )

    USER = 'SVC_PF_USER'
    ACCOUNT = 'xz30609.central-us.azure'
    WAREHOUSE = 'COMPUTE_WH'
    DATABASE = 'UNITEDCENTER_PROD_RAW_DB'
    SCHEMA = 'MICROSOFT_SECURITY'

    conn = snowflake.connector.connect(
    user=USER,
    private_key = p_key,
    account=ACCOUNT,
    warehouse=WAREHOUSE,
    database=DATABASE,
    schema=SCHEMA)
    conn.timeout = 6000   
    conn.autocommit = True

    dataToLoad = json.loads(data)
    dataToLoad = json.dumps(dataToLoad)
    # Call SP and trap Error if raised
    cursor = conn.cursor()
    # Execute Transactions load
    cursor.execute('CALL load_defender_data(%s)', dataToLoad)
    print('inserted data')    

    #except pyodbc.Error as err:
    #    print('Error !!!!! %s' % err)
    #except:
    #   print('something else failed miserably')

    conn.close()
    print('closed db connection')

@flow (retries = 3, retry_delay_seconds = 20, log_prints=True)
def main_flow():
    jsonData = asyncio.run(getData())
    print(jsonData)
    loadJson(jsonData)

if __name__ == '__main__':
    jsonData = asyncio.run(getData())
    print(type(jsonData))
    loadJson(jsonData)
m
thought for 25.8 seconds
The unexpected indentation is on the
print(out)
line inside
getData()
— it’s indented deeper than the surrounding lines and uses a tab while the rest of the function uses spaces. Problem area:
Copy code
async def getData():
    #Path to your PowerShell script
    powershell_script_path = "DefenderGetAlerts.ps1"

    out = await shell_run_command(..., return_all=True)
	print(out)   # <-- extra indent + tab here (inconsistent with spaces above)
    return str(out)
Fix by dedenting
print(out)
to the same level as the lines above/below and use spaces consistently:
Copy code
async def getData():
    # Path to your PowerShell script
    powershell_script_path = "DefenderGetAlerts.ps1"

    out = await shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1', shell="bash", return_all=True)
    print(out)
    return str(out)
Tip: mixing tabs and spaces can trigger “unexpected indent” or “inconsistent use of tabs and spaces” errors. Convert all indentation to 4 spaces and disable tabs in your editor.
c
@Marvin Now my .ps1 file is returning an empty string. The file's code is below:
Copy code
# Returns Alerts created in the past 48 hours.

$token = ./Get-Token.ps1       #run the script Get-Token.ps1  - make sure you are running this script from the same folder of Get-Token.ps1

# Get Alert from the last 48 hours. Make sure you have alerts in that time frame.
$dateTime = (Get-Date).ToUniversalTime().AddHours(-120).ToString("o")

# The URL contains the type of query and the time filter we create above
# Read more about [other query options and filters](get-alerts.md).
$url = "<https://api.securitycenter.microsoft.com/api/alerts>?`$filter=alertCreationTime ge $dateTime"

# Set the WebRequest headers
$headers = @{
    'Content-Type' = 'application/json'
    Accept = 'application/json'
    Authorization = "Bearer $token"
}

# Send the webrequest and get the results.
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop

# Extract the alerts from the results.
$alerts =  ($response.Content | ConvertFrom-Json).value | ConvertTo-Json -Compress

# Get string with the execution time. We concatenate that string to the output file to avoid overwrite the file
$dateTimeForFileName = Get-Date -Format o | foreach {$_ -replace ":", "."}

Write-Output $response
m
thought for 113.3 seconds
The empty string means your script didn’t write anything to stdout that Prefect could capture. In your current PS1, there are a few issues that can lead to that: What’s causing the empty output - You’re outputting the response object, not the parsed JSON: -
Write-Output $response
writes the WebResponseObject’s string representation (typically its Content). If the body is empty (e.g., 204 No Content) you’ll get an empty string. - You likely intended to output
$alerts
(the JSON you built), not
$response
. - Your URL still has Slack’s angle brackets: -
$url = "<https://api.securitycenter.microsoft.com/api/alerts>?..."
- The
<
and
>
should not be there; they can break the request and lead to unexpected behavior. Use the plain URL string. - Token capture can silently be empty: -
$token = ./Get-Token.ps1
will only capture success-stream output. If
Get-Token.ps1
uses
Write-Host
or prints extra text,
$token
may be empty/whitespace and the request won’t authenticate. Prefer the call operator and trim:
$token = (& ./Get-Token.ps1).Trim()
. A corrected PS1 that reliably emits one line of JSON
Copy code
$ErrorActionPreference = 'Stop'

# Capture only the token text from your token script
$token = (& ./Get-Token.ps1).Trim()
if ([string]::IsNullOrWhiteSpace($token)) { Write-Error "Empty token from Get-Token.ps1"; exit 1 }

# Last 48 hours (adjust as needed)
$dateTime = (Get-Date).ToUniversalTime().AddHours(-48).ToString("o")

# IMPORTANT: no angle brackets
$url = "<https://api/securitycenter.microsoft.com/api/alerts?`$filter=alertCreationTime> ge $dateTime"

$headers = @{
  'Content-Type' = 'application/json'
  Accept         = 'application/json'
  Authorization  = "Bearer $token"
}

# Prefer Invoke-RestMethod for JSON
$response = Invoke-RestMethod -Method Get -Uri $url -Headers $headers

# Normalize to an array and print a single-line JSON string
$alerts = $response.value
if ($null -eq $alerts) { $alerts = @() }
$alerts | ConvertTo-Json -Compress
If you want to keep Invoke-WebRequest:
Copy code
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop
$json = $response.Content | ConvertFrom-Json
$alerts = $json.value
if ($null -eq $alerts) { $alerts = @() }
$alerts | ConvertTo-Json -Compress
Debugging tips - Temporarily switch your Prefect call to capture all lines:
Copy code
out = await shell_run_command(
  command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1',
  shell='bash',
  return_all=True
)
print(out)
- Add quick checks in the script (use Write-Host so they don’t contaminate the final JSON line):
Copy code
Write-Host "token.len=$($token.Length)"
# For Invoke-WebRequest: Write-Host "status=$($response.StatusCode)"
- Ensure your URL string has no
<
or
>
anywhere. Reference for how Prefect captures output: - return_all=False → last stdout line as a string; no output → "" - return_all=True → list of all stdout lines prefect-shell
c
@Marvin Where is the indentation error in this code?
Copy code
import requests, json, pyodbc, pandas as pd, re
from prefect import flow, task
from azure.storage.blob import BlobClient, BlobServiceClient
from azure.identity import DefaultAzureCredential
from prefect_snowflake.credentials import SnowflakeCredentials
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.backends import default_backend
from prefect_shell import shell_run_command
import snowflake.connector
import subprocess
import asyncio

#Get Raw Data from API
@flow (retries = 0, retry_delay_seconds = 20)
async def getData():
    #Path to your PowerShell script
    powershell_script_path = "DefenderGetAlerts.ps1"

    out = await shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1', shell="bash", return_all=True)
    print(out)
    return out

#Load JSON data into Snowflake
@flow (retries = 0, retry_delay_seconds = 20, log_prints=True)
def loadJson(data):
    print("data is below:")
	print(data)
    blob_service_client = BlobServiceClient(
        account_url="<https://prefectscriptspy.blob.core.windows.net/>",
        credential='mec3I4AojCiv764tjydLxY8twl16IwIe+9iJVsWgQnG2V+aAsrEIhhcUozcMMRxVIsQKz43V82Un+ASttIsz3w==')
    
    container_name = "prefectscripts"
    blob_name = "rsa_key.p8" # e.g., "private_key.pem"

    container_client = blob_service_client.get_container_client(container_name)
    blob_client = container_client.get_blob_client(blob_name)

    try:
        download_stream = blob_client.download_blob()
        private_key_content = download_stream.readall()
        # If the private key is text-based (e.g., PEM), you might want to decode it
        # private_key_string = private_key_content.decode("utf-8")
        
        print("Private key retrieved successfully.")
        # Now, private_key_content holds the bytes of your private key
        # You can process it further, e.g., load it into a cryptographic library
    except Exception as e:
        print(f"Error retrieving private key: {e}")
    
    p_key = serialization.load_pem_private_key(
        private_key_content,
        password=None,  # Set to your password if the key is encrypted
        backend=default_backend()
        )

    USER = 'SVC_PF_USER'
    ACCOUNT = 'xz30609.central-us.azure'
    WAREHOUSE = 'COMPUTE_WH'
    DATABASE = 'UNITEDCENTER_PROD_RAW_DB'
    SCHEMA = 'MICROSOFT_SECURITY'

    conn = snowflake.connector.connect(
    user=USER,
    private_key = p_key,
    account=ACCOUNT,
    warehouse=WAREHOUSE,
    database=DATABASE,
    schema=SCHEMA)
    conn.timeout = 6000   
    conn.autocommit = True

    dataToLoad = json.loads(data)
    dataToLoad = json.dumps(dataToLoad)
    # Call SP and trap Error if raised
    cursor = conn.cursor()
    # Execute Transactions load
    cursor.execute('CALL load_defender_data(%s)', dataToLoad)
    print('inserted data')    

    #except pyodbc.Error as err:
    #    print('Error !!!!! %s' % err)
    #except:
    #   print('something else failed miserably')

    conn.close()
    print('closed db connection')

@flow (retries = 3, retry_delay_seconds = 20, log_prints=True)
def main_flow():
    jsonData = asyncio.run(getData())
    print(jsonData)
    loadJson(jsonData)

if __name__ == '__main__':
    jsonData = asyncio.run(getData())
    print(type(jsonData))
    loadJson(jsonData)
m
thought for 22.3 seconds
It’s on the
print(data)
line inside
loadJson()
— it’s indented one level too deep and with a tab, while surrounding lines use spaces. Problem area:
Copy code
def loadJson(data):
    print("data is below:")
	print(data)   # <-- extra indent + tab here
    blob_service_client = BlobServiceClient(...
Fix:
Copy code
def loadJson(data):
    print("data is below:")
    print(data)
    blob_service_client = BlobServiceClient(...
Tip: - Convert all indentation to spaces (4 spaces per level). - To find stray tabs quickly:
grep -nP '\t' your_file.py
- Python can also help:
python -m tabnanny your_file.py
c
@Marvin I got the following error running a flow. Can you explain it?
FileNotFoundError: [Errno 2] No such file or directory: 'powershell.exe'
m
thought for 167.4 seconds
Short version: your flow tried to launch PowerShell, but the runtime environment doesn’t have it (or it isn’t on PATH). Python raised FileNotFoundError when spawning the process. Why this happens with Prefect: - If you’re using the prefect-shell integration (e.g.,
shell_run_command
or
ShellOperation
), the default shell is: - Windows: PowerShell - Linux/macOS: bash - On Windows hosts/containers without PowerShell (or with a missing PATH), calling PowerShell fails with
FileNotFoundError: 'powershell.exe'
. How to fix it: - If you intended to use bash/sh, explicitly set the shell in your task:
Copy code
from prefect import flow
  from prefect_shell import shell_run_command

  @flow
  async def run():
      # Use bash on Unix-like systems
      await shell_run_command("echo hello", shell="bash")
      # Or sh
      # await shell_run_command("echo hello", shell="sh")
- If you want PowerShell, install it in the runtime image/host or use PowerShell 7 (
pwsh
) if that’s what you have:
Copy code
await shell_run_command("echo hello", shell="pwsh")
- If you’re running in Docker/Kubernetes, ensure the image includes the shell you’re targeting (PowerShell, bash, etc.) and that it’s on PATH. Could you share: - Where the flow is running (OS, container image, worker type)? - Whether you’re using
prefect-shell
(and the code snippet)? With that, I can give you a precise one-liner fix for your setup.
c
@Marvin Can you explain this error?
CreateProcessCommon:798: execvpe(/bin/bash) failed: No such file or director
m
thought for 41.1 seconds
That error means the process tried to launch bash at /bin/bash, but there’s no bash binary there (or on PATH). The OS call to exec the program failed, so you see execvpe(...): No such file or directory. Why you might see this: - Using prefect-shell with its default on non-Windows (bash), but your runtime doesn’t have bash (common in minimal images). - Running on Alpine Linux (which uses ash by default and often doesn’t include bash). - On Windows, explicitly setting
shell="bash"
without Git Bash or WSL bash available. How to fix: - Install bash in your image/host: - Debian/Ubuntu:
Copy code
apt-get update && apt-get install -y bash
- Alpine:
Copy code
apk add --no-cache bash
- RHEL/CentOS:
Copy code
yum install -y bash
- Or use a shell that exists: - On POSIX systems without bash:
Copy code
from prefect_shell import shell_run_command
    await shell_run_command("echo hello", shell="sh")
    # On Alpine you can also use shell="ash"
- On Windows:
Copy code
await shell_run_command("echo hello", shell="pwsh")  # PowerShell 7
    # or shell="powershell" if Windows PowerShell 5.x is installed
If you share where this is running (OS/container image/worker type) and how you’re calling the command, I can suggest the exact one-liner to fix it.
c
@Marvin I received this error. Can you explain it?
Copy code
RuntimeError: Command failed with exit code 127:
/tmp/prefect-4_uyidog: line 1: .\Get-Token-Intune.ps1: command not found
m
thought for 42.3 seconds
Short answer: your command was run in a POSIX shell (bash/sh), but it contains a PowerShell-style invocation. Exit code 127 means “command not found,” and
.\Get-Token-Intune.ps1
is a PowerShell path invocation that bash doesn’t understand. What’s happening - On non-Windows, Prefect’s shell helper defaults to
bash
unless you override it. - Your command
.\\Get-Token-Intune.ps1
is valid in PowerShell, not in bash. - bash tries to execute a command literally named
.\Get-Token-Intune.ps1
, can’t find it, and exits with 127. Fix options - Run with PowerShell instead of bash:
Copy code
from prefect import flow
  from prefect_shell import shell_run_command

  @flow
  async def run():
      # Windows PowerShell (5.x)
      await shell_run_command(r".\Get-Token-Intune.ps1", shell="powershell", cwd="/path/to/script")

      # PowerShell 7 (cross‑platform)
      await shell_run_command("./Get-Token-Intune.ps1", shell="pwsh", cwd="/path/to/script")
Notes: - Use
shell="powershell"
on Windows,
shell="pwsh"
if you have PowerShell 7 installed (Linux/macOS/Windows). - Set
cwd
to the directory containing the script or use an absolute path. - Or explicitly invoke PowerShell via the command:
Copy code
await shell_run_command("pwsh -File ./Get-Token-Intune.ps1", cwd="/path/to/script")
  # or on Windows:
  await shell_run_command("powershell -File .\\Get-Token-Intune.ps1", cwd="C:\\path\\to\\script")
If you meant to use bash instead, you’d need a bash-compatible script and syntax (PowerShell .ps1 files won’t run under bash). If you share: - OS/runtime (Windows/Linux/macOS, container image), - Whether you have PowerShell 7 (
pwsh
) or Windows PowerShell (
powershell
), - The code snippet calling the command, I can give you the precise one-liner that will work in your environment.
c
@Marvin How do I send print statements to my logs in Prefect?
m
thought for 136.6 seconds
Great question. In Prefect 3.x there are two ways to get “print-like” messages into your Prefect logs: 1) Easiest: turn on print capture - Prefect can intercept Python print output and forward it to your run logs. - Enable it per flow/task or globally: - Per flow/task:
Copy code
from prefect import flow, task

    @task(log_prints=True)
    def t():
        print("This will appear in Prefect logs")

    @flow(log_prints=True)
    def f():
        print("Flow prints are logged too")
        t()

    f()
- Globally (affects all flows/tasks):
Copy code
export PREFECT_LOGGING_LOG_PRINTS=True
- Notes: - Only prints to stdout/stderr are captured. - They are logged at INFO level. - Requires a running flow/task context. 2) Recommended: use the run logger - Prefer structured logging with
get_run_logger()
for reliability and control:
Copy code
from prefect import flow, task
  from prefect.logging import get_run_logger

  @task
  def t():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Task message")
    logger.warning("Heads up")
    logger.debug("Debug details")

  @flow
  def f():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Starting flow")
    t()
- Benefits: proper context (flow/task IDs), levels, formatting, and automatic sending to the Prefect API. Optional configuration you might find useful: - Set default logging level:
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
- Disable or tune API logging:
Copy code
export PREFECT_LOGGING_TO_API_ENABLED=True
  export PREFECT_LOGGING_TO_API_BATCH_INTERVAL=2.0
- Customize formatting/console color/markup via other
PREFECT_LOGGING_*
settings if needed. If you share how you’re running (local, Docker, worker type) I can suggest the minimal settings to get your prints into the UI with your setup.
c
@Marvin Can you explain the below error? I have all the files that I need to run this code in the same directory.
Copy code
Task run 'shell_run_command-406' - Task run failed with exception: FileNotFoundError(2, 'The system cannot find the file specified', None, 2, None)
m
thought for 62.4 seconds
That error means the OS could not start the process you asked Prefect to run. In practice, with
prefect-shell
this usually comes from one of these: - The shell executable is missing - Windows default is PowerShell. If
powershell.exe
or
pwsh.exe
isn’t installed or on PATH, process creation fails with FileNotFoundError. - On Windows, setting
shell="bash"
will also fail unless you actually have bash installed. - The working directory isn’t what you expect - Your files may be “in the same directory” as your code locally, but the flow run’s working directory can differ (e.g., in a worker, container, or deployment). Then relative paths won’t resolve. - The command references a file/executable that doesn’t exist at the given path in the run environment. What to try 1) Verify the shell exists and matches your environment - Windows PowerShell:
Copy code
await shell_run_command("echo hello", shell="powershell")
- PowerShell 7:
Copy code
await shell_run_command("echo hello", shell="pwsh")
- Linux/macOS:
Copy code
await shell_run_command("echo hello", shell="bash")  # or shell="sh"
If this simple echo fails, the chosen shell isn’t available. 2) Set the working directory explicitly or use absolute paths - Point
cwd
to the folder that contains your files:
Copy code
await shell_run_command("./my_script.ps1 arg1", shell="pwsh", cwd="C:\\path\\to\\dir")
  # or
  await shell_run_command("./my_script.sh arg1", shell="bash", cwd="/app/dir")
3) Match command syntax to the shell you’re using - PowerShell (Windows PowerShell or pwsh):
Copy code
await shell_run_command("./Get-Token-Intune.ps1", shell="pwsh", cwd="C:\\path\\to\\dir")
- Bash/sh:
Copy code
await shell_run_command("./script.sh", shell="bash", cwd="/path/to/dir")
  # or: await shell_run_command("bash script.sh", shell="bash", cwd="/path/to/dir")
If you can share: - OS/runtime (Windows/Linux/macOS, container image), - Your
shell_run_command
call (including
shell=
and
cwd=
), - The exact command you’re running, I can pinpoint the cause and give you the exact one-liner fix.
c
@Marvin How do I fix the following error?
Copy code
RuntimeError: Command failed with exit code 2:
./DefenderGetAlerts.ps1: line 3: =: command not found
./DefenderGetAlerts.ps1: line 6: syntax error near unexpected token `('
./DefenderGetAlerts.ps1: line 6: `$dateTime = (Get-Date).ToUniversalTime().AddHours(-48).ToString("o")'
The code block is:
Copy code
# Returns Alerts created in the past 48 hours.

$token = ./Get-Token.ps1       #run the script Get-Token.ps1  - make sure you are running this script from the same folder of Get-Token.ps1

# Get Alert from the last 48 hours. Make sure you have alerts in that time frame.
$dateTime = (Get-Date).ToUniversalTime().AddHours(-48).ToString("o")

# The URL contains the type of query and the time filter we create above
# Read more about [other query options and filters](get-alerts.md).
$url = "<https://api.securitycenter.microsoft.com/api/alerts>?`$filter=alertCreationTime ge $dateTime"

# Set the WebRequest headers
$headers = @{
    'Content-Type' = 'application/json'
    Accept = 'application/json'
    Authorization = "Bearer $token"
}

# Send the webrequest and get the results.
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop

# Extract the alerts from the results.
$alerts =  ($response | ConvertFrom-Json).value | ConvertTo-Json

# Get string with the execution time. We concatenate that string to the output file to avoid overwrite the file
$dateTimeForFileName = Get-Date -Format o | foreach {$_ -replace ":", "."}

Write-Output $alerts
m
thought for 128.6 seconds
You’re running a PowerShell script (.ps1) in a POSIX shell (bash/sh). The errors like “=: command not found” and “syntax error near unexpected token (‘” happen when bash tries to interpret PowerShell syntax. Exit code 2 is from the shell failing to parse/execute. Why this happens with Prefect: - In Prefect 3.x, the default shell is bash on non-Windows. If you don’t override it, PowerShell scripts are executed by bash and crash. - On Windows, default is PowerShell; on Linux/macOS you must use PowerShell 7 (
pwsh
) explicitly and have it installed. How to fix - Run the script with PowerShell, not bash, and set the working directory to where the files are: - Linux/macOS (requires PowerShell 7 installed):
Copy code
from prefect import flow
    from prefect_shell import shell_run_command

    @flow
    async def run():
        await shell_run_command(
            "./DefenderGetAlerts.ps1",
            shell="pwsh",
            cwd="/path/to/dir"
        )
- Windows:
Copy code
from prefect import flow
    from prefect_shell import shell_run_command

    @flow
    async def run():
        await shell_run_command(
            ".\\DefenderGetAlerts.ps1",
            shell="powershell",
            cwd="C:\\path\\to\\dir"
        )
- Or call PowerShell explicitly from the default bash shell:
Copy code
await shell_run_command("pwsh -File ./DefenderGetAlerts.ps1", cwd="/path/to/dir")
Make your script resilient to the working directory - Inside your .ps1, reference the sibling script via the script’s own directory:
Copy code
$token = & (Join-Path $PSScriptRoot 'Get-Token.ps1')
- Build the URL cleanly to avoid escaping confusion:
Copy code
$base = '<https://api.securitycenter.microsoft.com/api/alerts>'
  $url  = "$base?`$filter=alertCreationTime ge $dateTime"
If this still fails, please share: - OS/container image where the flow runs, and whether `pwsh`/`powershell` is installed - Your exact
shell_run_command(...)
call (including
shell=
and
cwd=
) - The path to the scripts at runtime I’ll tailor a one-liner that works in your environment.
c
@Marvin I'm getting the following error. Can you help explain it?
Copy code
FileNotFoundError: [WinError 2] The system cannot find the file specified
m
thought for 44.8 seconds
That Windows error means the OS couldn’t find the executable or file you asked it to run/open. In Prefect, this most often shows up when a shell command or script path doesn’t exist in the runtime environment or the working directory isn’t what you expect. Common causes and fixes: - Missing shell or wrong shell - If you’re using prefect-shell, make sure you’re calling a shell that exists. - Test quickly:
Copy code
await shell_run_command("echo hello", shell="powershell")
    # or if you have PowerShell 7:
    await shell_run_command("echo hello", shell="pwsh")
- If even echo fails, the chosen shell isn’t available on PATH. - Wrong working directory (relative paths don’t resolve) - The flow’s runtime CWD in a worker/container may differ from your local dev folder. - Set
cwd
explicitly or use absolute paths:
Copy code
await shell_run_command(".\\DefenderGetAlerts.ps1", shell="powershell", cwd="C:\\path\\to\\dir")
    # or
    await shell_run_command("pwsh -File ./DefenderGetAlerts.ps1", cwd="C:\\path\\to\\dir")
- File not included in the run environment - If this is a deployment, only files available in the runtime (e.g., your repo or image) are present. Local uncommitted files won’t be there. Ensure scripts/data are checked into the repo or baked into the image and pulled at run time. - Path/quoting issues on Windows - Paths with spaces must be quoted if you build commands yourself. - Prefer passing paths via
cwd
or use absolute paths. Quick debugging steps: - Print the current working directory and list files to confirm:
Copy code
await shell_run_command("Get-Location; Get-ChildItem", shell="powershell")
- Check a file exists before running:
Copy code
from pathlib import Path
  p = Path("DefenderGetAlerts.ps1")
  assert p.exists(), f"Missing: {p.resolve()}"
If you share: - How you’re invoking the command (your
shell_run_command(...)
call), - The OS/container image and whether you have
powershell
or
pwsh
, - Whether this is a local run or a deployment/worker, I’ll pinpoint the cause and give you the exact one-liner fix.
c
@Marvin Does Prefect utilize the service TruffleHog? Attached is the web page for the service
<https://trufflesecurity.com/trufflehog>
m
thought for 89.9 seconds
Short answer: No—Prefect does not use or integrate with TruffleHog. I searched the Prefect 3.x codebase, docs, dependencies, and community discussions and found no references to TruffleHog or any official integration. If you’re looking for secret scanning, Prefect itself focuses on: - Storing/securing credentials via Blocks and Variables - Masking sensitive values in logs - Integrations with cloud secret managers (e.g., AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) If you want to add TruffleHog to your workflow, you can run it as part of a CI step or invoke it from a Prefect flow using
prefect-shell
(e.g.,
shell_run_command("trufflehog ...")
) and capture the results. Happy to sketch a quick example based on your environment.
c
@Marvin Can you explain this error?
OSError: [WinError 193] %1 is not a valid Win32 application
m
thought for 61.0 seconds
Short explanation: Windows tried to “run” a file that isn’t a runnable Windows program for CreateProcess. WinError 193 (“%1 is not a valid Win32 application”) typically means you invoked a script or non‑Windows binary without the right interpreter/shell, passed a directory instead of an executable, or targeted a malformed/corrupted executable. Common causes in Prefect runs (especially with prefect-shell): - Running a script without the correct shell/interpreter - .ps1 needs PowerShell: use
shell="powershell"
(Windows) or
shell="pwsh"
(PowerShell 7 on any OS) - .bat/.cmd should be run by PowerShell (works) or via
cmd /c
- .py needs
python script.py
(Windows doesn’t honor shebangs) - Trying to run a non-Windows binary (e.g., Linux ELF) on Windows - Passing a path that’s a folder or wrong file - Shebang-only scripts (no extension) on Windows without an association How to fix (pick what fits your case): - PowerShell script
Copy code
from prefect import flow
  from prefect_shell import shell_run_command

  @flow
  async def run():
      await shell_run_command(".\\script.ps1", shell="powershell", cwd="C:\\path\\to\\dir")
      # or, with PowerShell 7 installed:
      # await shell_run_command("./script.ps1", shell="pwsh", cwd="C:\\path\\to\\dir")
- Batch script
Copy code
await shell_run_command(".\\script.bat", shell="powershell", cwd="C:\\path\\to\\dir")
  # or
  await shell_run_command("cmd /c script.bat", shell="powershell", cwd="C:\\path\\to\\dir")
- Python script
Copy code
await shell_run_command('python "C:\\path\\to\\script.py"', shell="powershell")
- Executable: ensure it’s a native Windows .exe and the path is correct
Copy code
await shell_run_command('"C:\\path\\to\\tool.exe" --help', shell="powershell")
Quick diagnostics: - Confirm the working directory and files:
Copy code
await shell_run_command("Get-Location; Get-ChildItem", shell="powershell")
- Verify interpreter availability:
Copy code
await shell_run_command("Get-Command powershell; Get-Command pwsh", shell="powershell")
- Check the target path in Python before running:
Copy code
from pathlib import Path
  p = Path("C:\\path\\to\\thing")
  print(p.resolve(), p.exists(), p.is_dir())
If you share: - The exact
shell_run_command(...)
call (command,
shell=
,
cwd=
), - What file you’re trying to run, - Where the flow runs (local vs worker, container image, OS), I’ll pinpoint the cause and give you the exact one-liner fix.
c
@Marvin Can you explain this error?
Copy code
Task run 'shell_run_command-677' - Task run failed with exception: FileNotFoundError(2, 'The system cannot find the file specified', None, 2, None)
m
thought for 32.8 seconds
Short version: Windows couldn’t start the process you asked it to run. FileNotFoundError(2) means “the system can’t find the file” — typically the shell or the target script/executable isn’t available at the path the run environment sees. Common causes with prefect-shell: - Missing shell: On Windows, Prefect defaults to PowerShell. If
powershell.exe
(or
pwsh
) isn’t installed/on PATH, process creation fails. - Wrong working directory: The flow’s runtime CWD (in a worker/container/deployment) may not be the folder you expect, so relative paths don’t resolve. - File not present in the runtime: The script/exe isn’t included in the image/repo that the deployment runs. - Path/quoting issues: Windows paths with spaces or backslashes can break if not quoted/escaped. How to fix: - Verify and choose the right shell:
Copy code
await shell_run_command("echo hello", shell="powershell")
  # or if you have PowerShell 7:
  await shell_run_command("echo hello", shell="pwsh")
If this fails, that shell isn’t available on PATH. - Set the working directory or use absolute paths:
Copy code
await shell_run_command(".\\DefenderGetAlerts.ps1", shell="powershell", cwd="C:\\path\\to\\dir")
  # or
  await shell_run_command("pwsh -File ./DefenderGetAlerts.ps1", cwd="C:\\path\\to\\dir")
- Match the command to the shell: - PowerShell script:
shell="powershell"
(Windows) or
shell="pwsh"
(PowerShell 7) - Batch:
cmd /c script.bat
(or run from PowerShell) - Python:
python path\\to\\script.py
(Windows doesn’t use shebangs) Quick diagnostics:
Copy code
# Check CWD and files (PowerShell)
await shell_run_command("Get-Location; Get-ChildItem", shell="powershell")

# Confirm your target exists before running (in Python)
from pathlib import Path
p = Path("C:\\path\\to\\your_file.ps1")
print(p.resolve(), p.exists(), p.is_file())
If you share: - Your exact
shell_run_command(...)
call (command,
shell=
,
cwd=
), - Where the flow runs (local vs worker, Docker image), - The file you’re trying to run, I can pinpoint the cause and give you the precise one-liner fix.
c
@Marvin The exact shell command is
out = await shell_run_command(command=f'".\\DefenderGetAlerts.ps1"', shell="pwsh", return_all=False)
m
thought for 52.5 seconds
Thanks for the exact command. Two likely issues here: 1) pwsh isn’t installed/on PATH - You set
shell="pwsh"
which is PowerShell 7. On many Windows hosts only Windows PowerShell 5.x (
powershell.exe
) is installed. - If
pwsh
is missing, Prefect can’t start the process and you’ll get
FileNotFoundError(2)
. - Fix: either install PowerShell 7, or switch to Windows PowerShell:
Copy code
out = await shell_run_command(
      command=".\\DefenderGetAlerts.ps1",
      shell="powershell",           # use this if pwsh is not available
      return_all=False
  )
To confirm availability:
Copy code
await shell_run_command("Get-Command pwsh", shell="powershell")
2) The command string is quoted, which prevents execution - You passed
command=f'".\\DefenderGetAlerts.ps1"'
(note the surrounding quotes). In PowerShell, a quoted path is just a string literal, not an invocation. - Use either the bare path or the call operator
&
when quoting: - Bare (no spaces in path):
Copy code
out = await shell_run_command(
        command=".\\DefenderGetAlerts.ps1",
        shell="pwsh",
        return_all=False
    )
- With quoting or spaces in the path:
Copy code
out = await shell_run_command(
        command='& ".\\DefenderGetAlerts.ps1"',
        shell="pwsh",
        return_all=False
    )
Also make sure the working directory is correct -
.\DefenderGetAlerts.ps1
is relative to the process CWD, which may not be your project folder in a worker/deployment. - Set
cwd
to the folder containing the script or use an absolute path:
Copy code
out = await shell_run_command(
      command='& ".\\DefenderGetAlerts.ps1"',
      shell="pwsh",
      cwd="C:\\path\\to\\dir",
      return_all=False
  )
Quick diagnostics - Check current directory and files:
Copy code
await shell_run_command("Get-Location; Get-ChildItem", shell="powershell")
- Verify the script exists from Python before calling:
Copy code
from pathlib import Path
  p = Path("C:\\path\\to\\dir\\DefenderGetAlerts.ps1")
  print(p.exists(), p.is_file())
If you confirm whether
pwsh
is available and where the script lives (and whether this is local or a deployment/worker), I can give you the exact one-liner that will work in your environment.