Charlie Nash
12/10/2025, 10:23 PMFileNotFoundError: [Errno 2] No such file or directory: 'powershell.exe'Marvin
12/10/2025, 10:24 PMMarvin
12/10/2025, 10:28 PMpowershell.exe, but the runtime can’t find it on the PATH (or it isn’t installed). This often happens when:
- You’re running on Linux/macOS (no Windows PowerShell), or
- You explicitly set shell="powershell.exe" (instead of powershell or pwsh), or
- You’re in a minimal Windows environment/container without PowerShell on PATH.
Quick ways to fix it
- If you’re on Linux/macOS and just need a shell:
- Use bash/sh explicitly.
- Example with prefect-shell:
pip install prefect-shell
from prefect import flow
from prefect_shell import shell_run_command
@flow
async def run():
out = await shell_run_command("echo hello", shell="bash")
print(out)
- If you need PowerShell cross‑platform:
- Install PowerShell Core (pwsh) and use shell="pwsh".
- Example:
from prefect import flow
from prefect_shell import shell_run_command
@flow
async def run():
out = await shell_run_command("Get-Process", shell="pwsh")
print(out)
- If you’re on Windows and want Windows PowerShell:
- Use shell="powershell" (do not use powershell.exe).
- Ensure C:\Windows\System32\WindowsPowerShell\v1.0\ is on PATH.
- Example:
from prefect import flow
from prefect_shell import shell_run_command
@flow
async def run():
out = await shell_run_command("Get-Process", shell="powershell")
print(out)
Notes
- Avoid .ps1 scripts on non‑Windows unless you have pwsh installed.
- If you’re calling subprocess yourself inside a task, swap ["powershell.exe", ...] for ["powershell", ...] on Windows or ["pwsh", ...] cross‑platform.
Can you share:
- Where this is running (OS / container image)?
- Whether you’re using prefect_shell (ShellOperation/shell_run_command) or subprocess?
- The code snippet where the shell is invoked?
With that, I can give a precise fix for your setup.Charlie Nash
12/15/2025, 4:06 PMMarvin
12/15/2025, 4:06 PMMarvin
12/15/2025, 4:09 PMpip install "prefect[shell]"
# or
pip install prefect-shell
2) Run a PowerShell command and return its output from your flow
from prefect import flow
from prefect_shell import shell_run_command
@flow
def run_powershell_inline():
# return_all=True -> capture full stdout as a list of lines
out = shell_run_command(
command='Write-Output "Hello from PowerShell"; Get-Date',
shell="powershell", # or "pwsh" for PowerShell 7+
return_all=True
)
return out
result = run_powershell_inline()
print(result)
Run an existing .ps1 script (with arguments)
from prefect import flow
from prefect_shell import shell_run_command
@flow
def run_ps_script(script_path: str, name: str):
# Use the call operator & to execute the script file
cmd = f'& "{script_path}" -Name "{name}"'
out = shell_run_command(
command=cmd,
shell="powershell", # or "pwsh"
return_all=True
)
return out
result = run_ps_script("C:\\scripts\\hello.ps1", "Marvin")
print(result)
Tips
- Choose your shell explicitly:
- shell="powershell" for Windows PowerShell (5.1)
- shell="pwsh" for PowerShell 7+ (cross‑platform)
- Return format:
- return_all=False (default) returns only the last line as a string
- return_all=True returns all stdout lines as a list[str]
- Non-zero exit codes raise a RuntimeError automatically (Prefect appends Exit $LastExitCode to propagate PS exit codes). Wrap in try/except if you want to handle failures:
from prefect import flow, get_run_logger
from prefect_shell import shell_run_command
@flow
def robust_run():
log = get_run_logger()
try:
return shell_run_command(command='throw "oops"', shell="powershell")
except RuntimeError as e:
log.error(f"PowerShell failed: {e}")
return None
- Working directory and environment:
shell_run_command(
command='Get-Item . | Select-Object -ExpandProperty FullName',
helper_command='cd C:\\Projects\\MyApp',
env={"APP_ENV": "prod"},
shell="powershell",
return_all=True,
)
- If you hit Execution Policy errors on Windows, set a process-scoped policy first:
shell_run_command(
command='Write-Output "Policy ok"; & "C:\\scripts\\doit.ps1"',
helper_command='Set-ExecutionPolicy -Scope Process Bypass -Force',
shell="powershell",
return_all=True,
)
Advanced: ShellOperation block
For multi-step or long-running scripts with streaming logs and reusability:
from prefect import flow
from prefect_shell import ShellOperation
@flow
def multi_step():
op = ShellOperation(
commands=[
'Write-Host "Step 1"',
'Write-Host "Step 2"',
],
shell="powershell",
stream_output=True
)
return op.run() # returns list[str]
Docs and references
- prefect-shell docs
- SDK reference
- Source on GitHub
- PyPI: prefect-shell
Would you like me to tailor an example to your exact setup? If you can share:
- Windows vs. Linux/macOS
- PowerShell version (powershell.exe vs pwsh)
- Whether you’re calling a .ps1 file and what arguments it needs
…I can provide a drop-in flow snippet.Charlie Nash
12/15/2025, 4:41 PM<coroutine object run_flow_async at 0x0000020BA3C595A0>Marvin
12/15/2025, 4:41 PMMarvin
12/15/2025, 4:43 PMfrom prefect import flow
from prefect_shell import shell_run_command
@flow
async def run_ps():
out = await shell_run_command(
command='Write-Output "Hello"',
shell="powershell",
return_all=True
)
return out
result = run_ps()
print(result)
2) Keep the flow sync by using the synchronous ShellOperation API
from prefect import flow
from prefect_shell import ShellOperation
@flow
def run_ps():
out = ShellOperation(
commands=['Write-Output "Hello"'],
shell="powershell"
).run()
return out
result = run_ps()
print(result)
3) If you used .submit() (i.e., ran it as a task), get the value with .result()
from prefect import flow
from prefect_shell import shell_run_command
@flow
def run_ps():
fut = shell_run_command.submit(
command='Write-Output "Hello"',
shell="powershell",
return_all=True
)
return fut.result()
print(run_ps())
If you share a small snippet of your flow, I can point out exactly where the missing await (or .result()) is.
Refs:
- prefect-shell docs
- prefect-shell APICharlie Nash
12/15/2025, 4:47 PMasync def getToken():
# Path to your PowerShell script
powershell_script_path = "Get-Token-Intune.ps1"
#command = ["powershell.exe", "-ExecutionPolicy", "Bypass", "-File", powershell_script_path]
#token = subprocess.run(command, capture_output=True, text=True)
out = await shell_run_command(powershell_script_path, shell="powershell", return_all=True)
return outMarvin
12/15/2025, 4:47 PMMarvin
12/15/2025, 4:48 PM& so it actually runs the script.
Here are two working patterns.
A) Keep it async and return a single string token
import asyncio
from prefect_shell import shell_run_command
async def getToken():
powershell_script_path = "C:\\path\\to\\Get-Token-Intune.ps1"
token = await shell_run_command(
command=f'& "{powershell_script_path}"', # invoke the script
shell="powershell", # or "pwsh"
return_all=False, # last line as a single string
# optional if you hit ExecutionPolicy issues:
helper_command='Set-ExecutionPolicy -Scope Process Bypass -Force'
)
return token.strip()
# If calling from plain Python (not inside an async Prefect flow)
if __name__ == "__main__":
print(asyncio.run(getToken()))
If you call getToken() from an async Prefect flow, make sure you await it:
from prefect import flow
@flow
async def my_flow():
token = await getToken()
return token
B) Make it synchronous using ShellOperation
from prefect_shell import ShellOperation
def getToken_sync():
powershell_script_path = "C:\\path\\to\\Get-Token-Intune.ps1"
lines = ShellOperation(
commands=[f'& "{powershell_script_path}"'],
shell="powershell",
stream_output=False,
).run()
return lines[-1].strip() # last line as your token
PowerShell output tips
- Make sure your .ps1 emits just the token on stdout. Prefer Write-Output $token (not Write-Host) for captured output.
- If the script writes multiple lines, return_all=False returns only the last line; or use return_all=True and pick the line you need: token = (await ...)[-1].strip().
If you can share how you’re calling getToken() (from a flow or a standalone script) and what the script prints, I can tailor this exactly.
Refs:
- prefect-shell docs
- prefect-shell APICharlie Nash
12/15/2025, 5:02 PM#Get Raw Data from API
@flow (retries = 0, retry_delay_seconds = 20)
async def getToken():
# Path to your PowerShell script
powershell_script_path = "Get-Token-Intune.ps1"
#command = ["powershell.exe", "-ExecutionPolicy", "Bypass", "-File", powershell_script_path]
#token = subprocess.run(command, capture_output=True, text=True)
out = await shell_run_command(powershell_script_path, shell="powershell", return_all=False)
return out.strip()Marvin
12/15/2025, 5:02 PMMarvin
12/15/2025, 5:03 PMout = await shell_run_command(
command=f'& "{powershell_script_path}"',
shell="powershell",
return_all=False
)
return out.strip()
2) Ensure your .ps1 emits the token to stdout
Prefect captures stdout. In PowerShell, prefer Write-Output $token or just $token as the last statement. Avoid:
- Write-Host (goes to the host/information stream, not stdout)
- `Write-Verbose`/`Write-Information` (not stdout)
- Out-Null (swallows output)
- Exit 0 (exits with success but no output)
Minimal example inside your .ps1:
# ... your logic to get $token ...
Write-Output $token
3) Inspect what the script prints
Temporarily capture all output lines so you can see what’s coming back:
lines = await shell_run_command(
command=f'& "{powershell_script_path}"',
shell="powershell",
return_all=True
)
print(lines) # for debugging
token = lines[-1].strip() if lines else ""
4) Common Windows gotchas
- If you hit execution policy issues, add a helper command:
out = await shell_run_command(
command=f'& "{powershell_script_path}"',
helper_command='Set-ExecutionPolicy -Scope Process Bypass -Force',
shell="powershell",
return_all=False
)
- Use an absolute path and quote it: C:\\path\\to\\Get-Token-Intune.ps1
- If your script relies on relative paths, set cwd or use helper_command='cd C:\\path\\to'
If you still get an empty string, can you share the last few lines of your PowerShell script (redacting secrets)? In particular, how the token is produced and printed. I can adjust the command or the script to ensure the token is captured.
Refs: prefect-shell docs | API referenceCharlie Nash
12/15/2025, 6:44 PMMarvin
12/15/2025, 6:44 PMMarvin
12/15/2025, 6:45 PM& and capture the last line.
Async flow using shell_run_command (returns a single string):
from prefect import flow
from prefect_shell import shell_run_command
@flow
async def fetch_token():
token = await shell_run_command(
command='& "C:\\path\\to\\Get-Token-Intune.ps1"', # or '& ".\\Get-Token-Intune.ps1"'
shell="powershell", # or "pwsh" for PowerShell 7+
return_all=False # last line as a single string
# helper_command='Set-ExecutionPolicy -Scope Process Bypass -Force', # if needed
# cwd="C:\\path\\to\\scripts", # if your script uses relative paths
)
return token.strip()
result = fetch_token()
print(result)
Synchronous alternative with ShellOperation:
from prefect import flow
from prefect_shell import ShellOperation
@flow
def fetch_token_sync():
lines = ShellOperation(
commands=['& "C:\\path\\to\\Get-Token-Intune.ps1"'],
shell="powershell",
stream_output=False
).run()
token = lines[-1].strip() if lines else ""
return token
print(fetch_token_sync())
If you still get an empty string:
- Double-check the script prints the token on stdout (Write-Output $token; not Write-Host).
- Temporarily debug with return_all=True to see everything the script emits:
lines = await shell_run_command(command='& ".\\Get-Token-Intune.ps1"', shell="powershell", return_all=True)
print(lines)
Docs:
- prefect-shell docs
- API referenceCharlie Nash
12/15/2025, 7:16 PMFileNotFoundError: [Errno 2] No such file or directory: 'powershell'Marvin
12/15/2025, 7:16 PMMarvin
12/15/2025, 7:20 PMpowershell binary). This commonly happens when:
- Using prefect-shell with shell="powershell" (or PowerShell-only commands)
- A library (e.g., Azure Identity) tries to fall back to PowerShell on Windows, but your flow actually runs in Linux
Ways to fix it (pick one):
1) If you’re using prefect-shell, run bash instead of PowerShell in Linux containers
- Update your flow to use bash-compatible commands:
from prefect_shell import shell_run_command
# Recommended when running in a Linux image
result = shell_run_command(command="echo hello", shell="bash")
- If you need cross-platform behavior, choose shell by platform:
import sys
from prefect_shell import shell_run_command
shell = "powershell" if sys.platform == "win32" else "bash"
result = shell_run_command(command="echo hello", shell=shell)
Reference: prefect-shell integration
2) If you truly need PowerShell, run on Windows or build an image that includes PowerShell
- Prefect’s official images are Linux and don’t include PowerShell. If you must use PS in a Linux container, install it and invoke pwsh (note: on Linux the executable is pwsh, not powershell, and exit code semantics differ).
Microsoft docs: Install PowerShell on Linux
3) If you’re using Azure Identity, disable its PowerShell credential
- In code:
from azure.identity import DefaultAzureCredential
cred = DefaultAzureCredential(exclude_powershell_credential=True)
- Or set env var:
AZURE_IDENTITY_DISABLE_POWERSHELL_CREDENTIAL=true
Docs: Azure Identity credential chain
A couple of clarifying questions so I can tailor the fix:
- Are you using prefect-shell anywhere (e.g., shell_run_command or ShellOperation)?
- Are you using Azure Identity or any library that might spawn PowerShell?
- What image are you running your flow in? If you can share your Dockerfile or the image tag and a short snippet of the code that runs a shell command, I can give you an exact patch.
General note: when you deploy flows to run from a Docker image in Prefect 3.x, they’ll typically run in Linux containers. Windows-specific commands (e.g., powershell, dir) will fail unless you target Windows infrastructure or adjust your commands. Deployment docs: Deploy flows with PrefectCharlie Nash
12/15/2025, 7:30 PM& ".\\Get-Token-Intune.ps1"\'\n')Marvin
12/15/2025, 7:30 PMMarvin
12/15/2025, 7:31 PM& ".\Get-Token-Intune.ps1".
- The & you see is just HTML-escaping of & — the real command was & ".\Get-Token-Intune.ps1".
- & and .\ are PowerShell syntax. But your flow ran in a Linux environment using bash (Prefect writes your command to a temp file like /tmp/prefect-xxxx and executes it with the chosen shell).
- Bash tried to parse the PowerShell command and failed with a syntax error. Exit code 2 is a bash “syntax misuse” error.
Why this happens in Prefect
- When using prefect-shell (e.g., shell_run_command or ShellOperation), the default shell is:
- PowerShell on Windows
- bash on Linux/macOS
- If you deploy to/run in a Linux container (Prefect’s default images are Linux), your PowerShell command gets executed by bash unless you explicitly run PowerShell.
How to fix
Pick one of these approaches:
1) Use bash-safe commands (recommended if you don’t need PowerShell)
- Replace PowerShell syntax with bash equivalents.
- Example:
from prefect_shell import shell_run_command
result = shell_run_command(command="echo hello", shell="bash")
2) If you need to run a PowerShell script, run PowerShell explicitly (and ensure it’s installed)
- In Linux, the executable is typically pwsh (PowerShell 7), not powershell.
- Update your command and/or shell:
from prefect_shell import shell_run_command
# Option A: keep bash as the shell, call pwsh directly
result = shell_run_command(
command='pwsh -NoProfile -File ./Get-Token-Intune.ps1',
shell='bash'
)
# Option B: use pwsh as the shell
result = shell_run_command(
command='./Get-Token-Intune.ps1',
shell='pwsh'
)
- Note the Linux-style path ./Get-Token-Intune.ps1 (not .\).
- Make sure your Docker image installs PowerShell:
- Install guide: Install PowerShell on Linux
3) If this script is only for getting tokens (e.g., Microsoft Intune/Azure), consider using the Python SDK directly to avoid spawning a shell
- For Azure authentication, azure-identity with DefaultAzureCredential often replaces custom PS scripts:
from azure.identity import DefaultAzureCredential
cred = DefaultAzureCredential()
Docs: Azure Identity credential chain
Helpful reference for shell tasks: prefect-shell integration
If you can share:
- The snippet where you call prefect-shell, and
- The base image/Dockerfile you’re using,
I can suggest the exact change (including whether to set shell="pwsh" or call pwsh -File ... directly).Charlie Nash
12/15/2025, 7:34 PMRuntimeError: Command failed with exit code 127:
/tmp/prefect-kq75xre5: line 1: .\Get-Token-Intune.ps1: command not foundMarvin
12/15/2025, 7:35 PMMarvin
12/15/2025, 7:35 PM.\Get-Token-Intune.ps1
That path is PowerShell/Windows syntax (.\). In a Linux container with bash:
- .\script.ps1 is not a valid command (hence 127)
- Bash can’t execute a PowerShell script unless you invoke PowerShell explicitly
- Even if the file is “in the same directory” on your machine, it must also exist in the container’s working directory at runtime
How to fix it
Pick one of these patterns:
1) Run the script with PowerShell (Linux) and use Linux-style paths
- Make sure PowerShell 7 is installed in the image (the executable is pwsh on Linux)
- Use ./Get-Token-Intune.ps1 (not .\...) and either call pwsh -File ... or set the shell to pwsh
Example A: keep bash as shell, call pwsh directly
from prefect_shell import shell_run_command
result = shell_run_command(
command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./Get-Token-Intune.ps1',
shell='bash', # your container’s default
cwd='/app' # set if needed
)
Example B: use pwsh as the shell
from prefect_shell import shell_run_command
result = shell_run_command(
command='./Get-Token-Intune.ps1',
shell='pwsh',
cwd='/app' # set if needed
)
PowerShell install guide (Linux): Install PowerShell on Linux
prefect-shell reference: prefect-shell integration
2) Ensure the script is actually present in the container and in the working directory
- If you’re using a Docker image, copy the script into it and set the working directory, e.g.:
WORKDIR /app
COPY Get-Token-Intune.ps1 /app/
- Or set cwd in shell_run_command to the directory where the script lives
- Quick sanity check inside the task:
shell_run_command(command='pwd && ls -la', shell='bash')
3) If you don’t need PowerShell, convert to a bash command or Python code
- For Azure tokens, the Python SDK can often replace PS scripts:
from azure.identity import DefaultAzureCredential
cred = DefaultAzureCredential()
Docs: Azure Identity credential chain
If you share:
- How you’re calling shell_run_command (snippet),
- Your Dockerfile or image tag,
I can pinpoint whether you need shell='pwsh', pwsh -File ..., cwd=..., or a Dockerfile change.Charlie Nash
12/15/2025, 7:46 PMRuntimeError: Command failed with exit code 2:
./Get-Token-Intune.ps1: line 4: =: command not found
./Get-Token-Intune.ps1: line 5: =: command not found
./Get-Token-Intune.ps1: line 6: =: command not found
./Get-Token-Intune.ps1: line 9: =: command not found
./Get-Token-Intune.ps1: line 10: =: command not found
./Get-Token-Intune.ps1: line 11: =: command not found
./Get-Token-Intune.ps1: line 12: resource: command not found
./Get-Token-Intune.ps1: line 13: client_id: command not found
./Get-Token-Intune.ps1: line 14: client_secret: command not found
./Get-Token-Intune.ps1: line 15: grant_type: command not found
./Get-Token-Intune.ps1: line 16: scope: command not found
./Get-Token-Intune.ps1: line 17: syntax error near unexpected token `}'
./Get-Token-Intune.ps1: line 17: `}'Marvin
12/15/2025, 7:46 PMMarvin
12/15/2025, 7:47 PM./Get-Token-Intune.ps1 without telling the system to use PowerShell (or without a PowerShell shebang), bash will try to parse the file. Lines like $var = ... or hashtables become “= …” and “resource:” to bash, which produces “command not found” and a final syntax error. Exit code 2 is bash’s “syntax misuse.”
How to fix it
Pick one of these:
1) Run the script with PowerShell (Linux) and use Linux-style paths
- Ensure PowerShell 7 is installed in your image (the binary is pwsh on Linux)
- Either set the Prefect shell to pwsh, or call pwsh -File ... from bash
Example A: use pwsh as the shell
from prefect_shell import shell_run_command
shell_run_command(
command='./Get-Token-Intune.ps1', # PowerShell will interpret this
shell='pwsh',
cwd='/app' # set if needed
)
Example B: keep bash as shell, invoke pwsh explicitly
from prefect_shell import shell_run_command
shell_run_command(
command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./Get-Token-Intune.ps1',
shell='bash',
cwd='/app'
)
2) Add a PowerShell shebang to the script so the OS picks the right interpreter
- At the top of Get-Token-Intune.ps1:
#!/usr/bin/env pwsh
- Make it executable:
chmod +x Get-Token-Intune.ps1
- Then ./Get-Token-Intune.ps1 will run under PowerShell even when launched from bash.
3) If you don’t need PowerShell, convert the logic to bash or Python (e.g., Azure auth via azure-identity) to avoid spawning PowerShell.
Also check:
- Use ./Get-Token-Intune.ps1 (Linux style), not .\...
- Ensure the script is actually in the container’s working directory (set cwd or verify with pwd && ls -la)
- Make sure the file has LF line endings; CRLF can cause odd shell parsing issues. If needed: dos2unix Get-Token-Intune.ps1
References:
- prefect-shell integration: prefect-shell
- Install PowerShell on Linux: Microsoft docs
If you share your shell_run_command call and Dockerfile/image, I can recommend the exact change (pwsh vs pwsh -File, cwd, permissions).Charlie Nash
12/15/2025, 8:24 PMawait shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./Get-Token-Intune.ps1', shell="bash", return_all=False)Charlie Nash
12/15/2025, 8:25 PMawait shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./Get-Token-Intune.ps1', shell="bash", return_all=False)
and my docker image is:
docker image is: FROM prefecthq/prefect:3.2.14-python3.9
COPY . /opt/prefect/Scripts/
WORKDIR /opt/prefect/Scripts/
RUN pip install --no-cache-dir prefect-snowflake GitPython PyGithub "prefect[shell]" "snowflake-connector-python[pandas]==4.0.0" "pandas==2.3.3" "numpy>=2.0,<3" "pyarrow==21.0.0" "prefect[shell]" adlfs s3fs requests pyodbc prefect_azure && apt-get update && apt-get install -y unixodbc -y curl ca-certificates && curl -sSL -o /usr/local/bin/kubectl <https://dl.k8s.io/release/v1.30.0/bin/linux/amd64/kubectl> && chmod +x /usr/local/bin/kubectl && apt-get update && apt-get install -y wget software-properties-common && wget -q "<https://packages.microsoft.com/config/debian/$(lsb_release> -rs)/packages-microsoft-prod.deb" -O packages-microsoft-prod.deb && dpkg -i packages-microsoft-prod.deb && rm packages-microsoft-prod.deb && apt-get updateMarvin
12/15/2025, 8:25 PMMarvin
12/15/2025, 8:26 PMpwsh -NoProfile -ExecutionPolicy Bypass -File ./Get-Token-Intune.ps1 with shell="bash". That’s good — bash will spawn pwsh to run the script.
- In your Dockerfile you add Microsoft’s repo (via packages-microsoft-prod.deb) but you never install PowerShell. There’s no apt-get install -y powershell, so pwsh won’t exist in the container.
- Minor: your snippet shows HTML-escaped characters (&&, >, <). Make sure your real Dockerfile uses &&, >=, < etc.
What to change in your Dockerfile
Add an explicit install of PowerShell (pwsh). Also, avoid relying on lsb_release and use the codename from /etc/os-release to pick the right Microsoft repo:
FROM prefecthq/prefect:3.2.14-python3.9
# Copy your code
COPY . /opt/prefect/Scripts/
WORKDIR /opt/prefect/Scripts/
# Python deps (dedupe prefect[shell] if you had it twice; optionally use prefect-shell explicitly)
RUN pip install --no-cache-dir \
prefect-snowflake \
GitPython \
PyGithub \
"prefect[shell]" \
"snowflake-connector-python[pandas]==4.0.0" \
"pandas==2.3.3" \
"numpy>=2.0,<3" \
"pyarrow==21.0.0" \
adlfs s3fs requests pyodbc prefect_azure
# OS deps + PowerShell
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates curl wget gnupg software-properties-common \
unixodbc \
&& . /etc/os-release \
&& wget -q "<https://packages.microsoft.com/config/debian/${VERSION_CODENAME}/packages-microsoft-prod.deb>" -O packages-microsoft-prod.deb \
&& dpkg -i packages-microsoft-prod.deb \
&& rm packages-microsoft-prod.deb \
&& apt-get update \
&& apt-get install -y --no-install-recommends powershell \
&& curl -sSL -o /usr/local/bin/kubectl <https://dl.k8s.io/release/v1.30.0/bin/linux/amd64/kubectl> \
&& chmod +x /usr/local/bin/kubectl \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
Sanity checks (run once in your flow or container)
- Verify PowerShell is present and in PATH:
await shell_run_command(command='which pwsh && pwsh -v', shell='bash')
- Verify the script exists where you expect:
await shell_run_command(command='pwd && ls -la', shell='bash', return_all=True)
- Then run your script (what you already have is fine):
await shell_run_command(
command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./Get-Token-Intune.ps1',
shell='bash',
return_all=False,
cwd='/opt/prefect/Scripts' # optional, but removes any doubt about CWD
)
A few additional tips
- Use Linux paths (./Get-Token-Intune.ps1, not .\...).
- Ensure the script has LF line endings (not CRLF). If needed during build:
- Install dos2unix: apt-get install -y dos2unix then dos2unix /opt/prefect/Scripts/Get-Token-Intune.ps1
- You don’t need the script to be executable when using pwsh -File, but it must be readable and present in the working directory.
- If you prefer, you can also run with shell="pwsh" and command='./Get-Token-Intune.ps1' once pwsh is installed.
References:
- prefect-shell: prefect-shell integration
- Install PowerShell on Linux: Microsoft docs
If this still errors after adding apt-get install -y powershell, share the new error text and I’ll help debug further.Charlie Nash
12/15/2025, 9:23 PMimport requests, json, pyodbc, pandas as pd, re
from prefect import flow, task
from azure.storage.blob import BlobClient, BlobServiceClient
from azure.identity import DefaultAzureCredential
from prefect_snowflake.credentials import SnowflakeCredentials
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.backends import default_backend
from prefect_shell import shell_run_command
#from prefect.filesystems import Azure
import snowflake.connector
import subprocess
import asyncio
#Get Raw Data from API
@flow (retries = 0, retry_delay_seconds = 20)
async def getData():
# Path to your PowerShell script
powershell_script_path = "DefenderGetAlerts.ps1"
out = await shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1', shell="bash", return_all=False)
return out.strip()
#Load JSON data into Snowflake
@flow (retries = 0, retry_delay_seconds = 20, log_prints=True)
def loadJson(data):
print(data)
blob_service_client = BlobServiceClient(
account_url="<https://prefectscriptspy.blob.core.windows.net/>",
credential='mec3I4AojCiv764tjydLxY8twl16IwIe+9iJVsWgQnG2V+aAsrEIhhcUozcMMRxVIsQKz43V82Un+ASttIsz3w==')
container_name = "prefectscripts"
blob_name = "rsa_key.p8" # e.g., "private_key.pem"
container_client = blob_service_client.get_container_client(container_name)
blob_client = container_client.get_blob_client(blob_name)
try:
download_stream = blob_client.download_blob()
private_key_content = download_stream.readall()
# If the private key is text-based (e.g., PEM), you might want to decode it
# private_key_string = private_key_content.decode("utf-8")
print("Private key retrieved successfully.")
# Now, private_key_content holds the bytes of your private key
# You can process it further, e.g., load it into a cryptographic library
except Exception as e:
print(f"Error retrieving private key: {e}")
p_key = serialization.load_pem_private_key(
private_key_content,
password=None, # Set to your password if the key is encrypted
backend=default_backend()
)
USER = 'SVC_PF_USER'
ACCOUNT = 'xz30609.central-us.azure'
WAREHOUSE = 'COMPUTE_WH'
DATABASE = 'UNITEDCENTER_PROD_RAW_DB'
SCHEMA = 'MICROSOFT_SECURITY'
conn = snowflake.connector.connect(
user=USER,
private_key = p_key,
account=ACCOUNT,
warehouse=WAREHOUSE,
database=DATABASE,
schema=SCHEMA)
conn.timeout = 6000
conn.autocommit = True
dataToLoad = json.loads(data)
dataToLoad = json.dumps(dataToLoad)
# Call SP and trap Error if raised
cursor = conn.cursor()
# Execute Transactions load
cursor.execute('CALL load_defender_data(%s)', dataToLoad)
print('inserted data')
#except pyodbc.Error as err:
# print('Error !!!!! %s' % err)
#except:
# print('something else failed miserably')
conn.close()
print('closed db connection')
@flow (retries = 3, retry_delay_seconds = 20, log_prints=True)
def main_flow():
jsonData = asyncio.run(getData())
print(jsonData)
loadJson(jsonData)
if __name__ == '__main__':
jsonData = asyncio.run(getData())
print(type(jsonData))
loadJson(jsonData)Marvin
12/15/2025, 9:23 PMMarvin
12/15/2025, 9:24 PMcursor = conn.cursor()
# Execute Transactions load <-- extra indent here
cursor.execute('CALL load_defender_data(%s)', dataToLoad)
Dedent that comment so it aligns with the surrounding lines:
cursor = conn.cursor()
# Execute Transactions load
cursor.execute('CALL load_defender_data(%s)', dataToLoad)
Also double-check there aren’t any stray tabs mixed with spaces around that spot. If you still get an IndentationError, please share the full traceback with the reported line number and I’ll pinpoint it exactly.Charlie Nash
12/15/2025, 9:36 PMout = await shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1', shell="bash", return_all=False)
And my .ps1 file is
# Returns Alerts created in the past 48 hours.
$token = ./Get-Token.ps1 #run the script Get-Token.ps1 - make sure you are running this script from the same folder of Get-Token.ps1
# Get Alert from the last 48 hours. Make sure you have alerts in that time frame.
$dateTime = (Get-Date).ToUniversalTime().AddHours(-120).ToString("o")
# The URL contains the type of query and the time filter we create above
# Read more about [other query options and filters](get-alerts.md).
$url = "<https://api.securitycenter.microsoft.com/api/alerts>?`$filter=alertCreationTime ge $dateTime"
# Set the WebRequest headers
$headers = @{
'Content-Type' = 'application/json'
Accept = 'application/json'
Authorization = "Bearer $token"
}
# Send the webrequest and get the results.
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop
# Extract the alerts from the results.
$alerts = ($response | ConvertFrom-Json).value | ConvertTo-Json
# Get string with the execution time. We concatenate that string to the output file to avoid overwrite the file
$dateTimeForFileName = Get-Date -Format o | foreach {$_ -replace ":", "."}
Write-Output $alerts
return $alertsMarvin
12/15/2025, 9:36 PMMarvin
12/15/2025, 9:38 PM# Returns Alerts created in the past 48 hours.
$token = ./Get-Token.ps1
# last 48 hours
$dateTime = (Get-Date).ToUniversalTime().AddHours(-48).ToString("o")
$url = "<https://api.securitycenter.microsoft.com/api/alerts?`$filter=alertCreationTime> ge $dateTime"
$headers = @{
'Content-Type' = 'application/json'
Accept = 'application/json'
Authorization = "Bearer $token"
}
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop
$alerts = ($response.Content | ConvertFrom-Json).value
if ($null -eq $alerts) { $alerts = @() } # ensure array instead of $null
$alertsJson = $alerts | ConvertTo-Json -Compress
Write-Output $alertsJson
# no 'return' needed; the last expression or Write-Output sends to stdout
Then your Python call can stay as-is:
out = await shell_run_command(
command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1',
shell='bash',
return_all=False
)
# out will be a single-line JSON string: [] or [{"...": "..."}]
2) Capture all lines from the script and join in Python
- Keep your current script, but change return_all to True:
lines = await shell_run_command(
command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1',
shell='bash',
return_all=True
)
out = "\n".join(lines) # reconstruct the full JSON
Additional tips
- If Get-Token.ps1 prints logging/status lines, those can become the “last line” and displace your JSON. Ensure it only writes the token to stdout, or assign its result with: $token = & ./Get-Token.ps1 and keep that script quiet except for the token.
- If there are no alerts, you’ll get [] (after the $null → @() coercion above) instead of null.
- You can sanity-check what Prefect sees with:
await shell_run_command(command='pwsh -c "Write-Output hello"', shell='bash', return_all=True)
If you share what exact string you get in Python (e.g., printing repr(out)), I can confirm whether it’s an empty string, the literal "null", or a truncated last line, and tailor the fix.Charlie Nash
12/15/2025, 9:51 PM['null', 'null']
Updated .ps1 script is:
# Returns Alerts created in the past 48 hours.
$token = ./Get-Token.ps1 #run the script Get-Token.ps1 - make sure you are running this script from the same folder of Get-Token.ps1
# Get Alert from the last 48 hours. Make sure you have alerts in that time frame.
$dateTime = (Get-Date).ToUniversalTime().AddHours(-120).ToString("o")
# The URL contains the type of query and the time filter we create above
# Read more about [other query options and filters](get-alerts.md).
$url = "<https://api.securitycenter.microsoft.com/api/alerts>?`$filter=alertCreationTime ge $dateTime"
# Set the WebRequest headers
$headers = @{
'Content-Type' = 'application/json'
Accept = 'application/json'
Authorization = "Bearer $token"
}
# Send the webrequest and get the results.
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop
# Extract the alerts from the results.
$alerts = ($response | ConvertFrom-Json).value | ConvertTo-Json -Compress
# Get string with the execution time. We concatenate that string to the output file to avoid overwrite the file
$dateTimeForFileName = Get-Date -Format o | foreach {$_ -replace ":", "."}
Write-Output $alerts
return $alertsMarvin
12/15/2025, 9:51 PMMarvin
12/15/2025, 9:52 PM$ErrorActionPreference = 'Stop'
$token = & ./Get-Token.ps1 # run script and capture only its output
# last 48 hours
$dateTime = (Get-Date).ToUniversalTime().AddHours(-48).ToString("o")
$url = "<https://api.securitycenter.microsoft.com/api/alerts?`$filter=alertCreationTime> ge $dateTime"
$headers = @{
'Content-Type' = 'application/json'
Accept = 'application/json'
Authorization = "Bearer $token"
}
# Prefer Invoke-RestMethod for JSON
$response = Invoke-RestMethod -Method Get -Uri $url -Headers $headers
$alerts = $response.value
if ($null -eq $alerts) { $alerts = @() } # ensure array instead of $null
# Emit a single line so Prefect can capture it as one item
$alerts | ConvertTo-Json -Compress
Then in Python:
- If you want just one string line:
out = await shell_run_command(
command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1',
shell='bash',
return_all=False
)
# out is a single-line JSON string; parse if needed:
# data = json.loads(out)
- If you keep return_all=True, you should now get ['[...]'] with one element; you can join if needed:
lines = await shell_run_command(..., return_all=True)
out = "\n".join(lines)
Why this works:
- No “return” duplicates the output
- Using Invoke-RestMethod or $response.Content ensures you actually parse JSON
- -Compress keeps JSON to a single line so return_all=False gives you the whole payload
If you still see "null", print the raw response for a quick check:
pwsh -NoProfile -Command "Invoke-WebRequest -Uri '<your url>' -Headers <headers> | Select-Object -ExpandProperty Content | Set-Content -Path debug.json"
or switch to Invoke-RestMethod and inspect $response directly.Charlie Nash
12/15/2025, 9:57 PM# Returns Alerts created in the past 48 hours.
$token = ./Get-Token.ps1 #run the script Get-Token.ps1 - make sure you are running this script from the same folder of Get-Token.ps1
# Get Alert from the last 48 hours. Make sure you have alerts in that time frame.
$dateTime = (Get-Date).ToUniversalTime().AddHours(-120).ToString("o")
# The URL contains the type of query and the time filter we create above
# Read more about [other query options and filters](get-alerts.md).
$url = "<https://api.securitycenter.microsoft.com/api/alerts>?`$filter=alertCreationTime ge $dateTime"
# Set the WebRequest headers
$headers = @{
'Content-Type' = 'application/json'
Accept = 'application/json'
Authorization = "Bearer $token"
}
# Send the webrequest and get the results.
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop
# Extract the alerts from the results.
$alerts = ($response.Content | ConvertFrom-Json).value | ConvertTo-Json -Compress
# Get string with the execution time. We concatenate that string to the output file to avoid overwrite the file
$dateTimeForFileName = Get-Date -Format o | foreach {$_ -replace ":", "."}
Write-Output $alertsMarvin
12/15/2025, 9:57 PMMarvin
12/15/2025, 9:58 PM$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop
$alerts = ($response.Content | ConvertFrom-Json).value | ConvertTo-Json -Compress
Write-Output $alerts
- If ($response.Content | ConvertFrom-Json).value is $null (e.g., no results, invalid token, or unexpected JSON shape), then piping $null to ConvertTo-Json yields the string null.
- Prefect’s shell_run_command(return_all=False) returns the last stdout line; if that’s null, that’s what you’ll get.
Common reasons .value is $null
- The Defender API returned an error or a different JSON shape because the token is missing/invalid. A common cause is Get-Token.ps1 printing via Write-Host (host stream), which won’t flow into $token = ./Get-Token.ps1. Use the call operator and ensure it writes to the success stream: $token = (& ./Get-Token.ps1).Trim().
- The URL string includes angle brackets (e.g., "<https://...>" from Slack). Those < and > should not be in the URL.
- The time filter returns no alerts and the API payload doesn’t include value (or it’s null). Some APIs return { value: [] }, others may omit value.
How to fix and make output predictable
- Prefer Invoke-RestMethod to get parsed JSON automatically and coerce nulls to an empty array before converting to JSON:
$ErrorActionPreference = 'Stop'
$token = (& ./Get-Token.ps1).Trim()
if ([string]::IsNullOrWhiteSpace($token)) { Write-Error "Empty token"; exit 1 }
# last 48 hours
$dateTime = (Get-Date).ToUniversalTime().AddHours(-48).ToString("o")
$url = "<https://api.securitycenter.microsoft.com/api/alerts?`$filter=alertCreationTime> ge $dateTime"
$headers = @{
'Content-Type' = 'application/json'
Accept = 'application/json'
Authorization = "Bearer $token"
}
# Parse JSON automatically
$response = Invoke-RestMethod -Method Get -Uri $url -Headers $headers
$alerts = $response.value
if ($null -eq $alerts) { $alerts = @() } # avoid null → "null"
# Emit a single line of JSON
$alerts | ConvertTo-Json -Compress
- If you keep Invoke-WebRequest, be sure to parse .Content (you are), then:
$json = $response.Content | ConvertFrom-Json
$alerts = $json.value
if ($null -eq $alerts) { $alerts = @() }
$alerts | ConvertTo-Json -Compress
- Make sure the URL has no < or `>`:
$url = "<https://api.securitycenter.microsoft.com/api/alerts?`$filter=alertCreationTime> ge $dateTime"
- Ensure Get-Token.ps1 outputs only the token via success output (Write-Output or a bare expression), not Write-Host. In the caller, use the call operator:
$token = (& ./Get-Token.ps1).Trim()
- For debugging, temporarily print what you’re actually getting:
Write-Output "Token length: $($token.Length)"
Write-Output ($response | ConvertTo-Json -Compress)
Once $alerts is an array (even empty), Prefect will capture valid JSON instead of the string null.Charlie Nash
12/15/2025, 10:04 PMimport requests, json, pyodbc, pandas as pd, re
from prefect import flow, task
from azure.storage.blob import BlobClient, BlobServiceClient
from azure.identity import DefaultAzureCredential
from prefect_snowflake.credentials import SnowflakeCredentials
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.backends import default_backend
from prefect_shell import shell_run_command
import snowflake.connector
import subprocess
import asyncio
#Get Raw Data from API
@flow (retries = 0, retry_delay_seconds = 20)
async def getData():
#Path to your PowerShell script
powershell_script_path = "DefenderGetAlerts.ps1"
out = await shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1', shell="bash", return_all=True)
print(out)
return str(out)
#Load JSON data into Snowflake
@flow (retries = 0, retry_delay_seconds = 20, log_prints=True)
def loadJson(data):
print(data)
blob_service_client = BlobServiceClient(
account_url="<https://prefectscriptspy.blob.core.windows.net/>",
credential='mec3I4AojCiv764tjydLxY8twl16IwIe+9iJVsWgQnG2V+aAsrEIhhcUozcMMRxVIsQKz43V82Un+ASttIsz3w==')
container_name = "prefectscripts"
blob_name = "rsa_key.p8" # e.g., "private_key.pem"
container_client = blob_service_client.get_container_client(container_name)
blob_client = container_client.get_blob_client(blob_name)
try:
download_stream = blob_client.download_blob()
private_key_content = download_stream.readall()
# If the private key is text-based (e.g., PEM), you might want to decode it
# private_key_string = private_key_content.decode("utf-8")
print("Private key retrieved successfully.")
# Now, private_key_content holds the bytes of your private key
# You can process it further, e.g., load it into a cryptographic library
except Exception as e:
print(f"Error retrieving private key: {e}")
p_key = serialization.load_pem_private_key(
private_key_content,
password=None, # Set to your password if the key is encrypted
backend=default_backend()
)
USER = 'SVC_PF_USER'
ACCOUNT = 'xz30609.central-us.azure'
WAREHOUSE = 'COMPUTE_WH'
DATABASE = 'UNITEDCENTER_PROD_RAW_DB'
SCHEMA = 'MICROSOFT_SECURITY'
conn = snowflake.connector.connect(
user=USER,
private_key = p_key,
account=ACCOUNT,
warehouse=WAREHOUSE,
database=DATABASE,
schema=SCHEMA)
conn.timeout = 6000
conn.autocommit = True
dataToLoad = json.loads(data)
dataToLoad = json.dumps(dataToLoad)
# Call SP and trap Error if raised
cursor = conn.cursor()
# Execute Transactions load
cursor.execute('CALL load_defender_data(%s)', dataToLoad)
print('inserted data')
#except pyodbc.Error as err:
# print('Error !!!!! %s' % err)
#except:
# print('something else failed miserably')
conn.close()
print('closed db connection')
@flow (retries = 3, retry_delay_seconds = 20, log_prints=True)
def main_flow():
jsonData = asyncio.run(getData())
print(jsonData)
loadJson(jsonData)
if __name__ == '__main__':
jsonData = asyncio.run(getData())
print(type(jsonData))
loadJson(jsonData)Marvin
12/15/2025, 10:04 PMMarvin
12/15/2025, 10:05 PMprint(out) line inside getData() — it’s indented deeper than the surrounding lines and uses a tab while the rest of the function uses spaces.
Problem area:
async def getData():
#Path to your PowerShell script
powershell_script_path = "DefenderGetAlerts.ps1"
out = await shell_run_command(..., return_all=True)
print(out) # <-- extra indent + tab here (inconsistent with spaces above)
return str(out)
Fix by dedenting print(out) to the same level as the lines above/below and use spaces consistently:
async def getData():
# Path to your PowerShell script
powershell_script_path = "DefenderGetAlerts.ps1"
out = await shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1', shell="bash", return_all=True)
print(out)
return str(out)
Tip: mixing tabs and spaces can trigger “unexpected indent” or “inconsistent use of tabs and spaces” errors. Convert all indentation to 4 spaces and disable tabs in your editor.Charlie Nash
12/15/2025, 10:07 PM# Returns Alerts created in the past 48 hours.
$token = ./Get-Token.ps1 #run the script Get-Token.ps1 - make sure you are running this script from the same folder of Get-Token.ps1
# Get Alert from the last 48 hours. Make sure you have alerts in that time frame.
$dateTime = (Get-Date).ToUniversalTime().AddHours(-120).ToString("o")
# The URL contains the type of query and the time filter we create above
# Read more about [other query options and filters](get-alerts.md).
$url = "<https://api.securitycenter.microsoft.com/api/alerts>?`$filter=alertCreationTime ge $dateTime"
# Set the WebRequest headers
$headers = @{
'Content-Type' = 'application/json'
Accept = 'application/json'
Authorization = "Bearer $token"
}
# Send the webrequest and get the results.
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop
# Extract the alerts from the results.
$alerts = ($response.Content | ConvertFrom-Json).value | ConvertTo-Json -Compress
# Get string with the execution time. We concatenate that string to the output file to avoid overwrite the file
$dateTimeForFileName = Get-Date -Format o | foreach {$_ -replace ":", "."}
Write-Output $responseMarvin
12/15/2025, 10:07 PMMarvin
12/15/2025, 10:09 PMWrite-Output $response writes the WebResponseObject’s string representation (typically its Content). If the body is empty (e.g., 204 No Content) you’ll get an empty string.
- You likely intended to output $alerts (the JSON you built), not $response.
- Your URL still has Slack’s angle brackets:
- $url = "<https://api.securitycenter.microsoft.com/api/alerts>?..."
- The < and > should not be there; they can break the request and lead to unexpected behavior. Use the plain URL string.
- Token capture can silently be empty:
- $token = ./Get-Token.ps1 will only capture success-stream output. If Get-Token.ps1 uses Write-Host or prints extra text, $token may be empty/whitespace and the request won’t authenticate. Prefer the call operator and trim: $token = (& ./Get-Token.ps1).Trim().
A corrected PS1 that reliably emits one line of JSON
$ErrorActionPreference = 'Stop'
# Capture only the token text from your token script
$token = (& ./Get-Token.ps1).Trim()
if ([string]::IsNullOrWhiteSpace($token)) { Write-Error "Empty token from Get-Token.ps1"; exit 1 }
# Last 48 hours (adjust as needed)
$dateTime = (Get-Date).ToUniversalTime().AddHours(-48).ToString("o")
# IMPORTANT: no angle brackets
$url = "<https://api/securitycenter.microsoft.com/api/alerts?`$filter=alertCreationTime> ge $dateTime"
$headers = @{
'Content-Type' = 'application/json'
Accept = 'application/json'
Authorization = "Bearer $token"
}
# Prefer Invoke-RestMethod for JSON
$response = Invoke-RestMethod -Method Get -Uri $url -Headers $headers
# Normalize to an array and print a single-line JSON string
$alerts = $response.value
if ($null -eq $alerts) { $alerts = @() }
$alerts | ConvertTo-Json -Compress
If you want to keep Invoke-WebRequest:
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop
$json = $response.Content | ConvertFrom-Json
$alerts = $json.value
if ($null -eq $alerts) { $alerts = @() }
$alerts | ConvertTo-Json -Compress
Debugging tips
- Temporarily switch your Prefect call to capture all lines:
out = await shell_run_command(
command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1',
shell='bash',
return_all=True
)
print(out)
- Add quick checks in the script (use Write-Host so they don’t contaminate the final JSON line):
Write-Host "token.len=$($token.Length)"
# For Invoke-WebRequest: Write-Host "status=$($response.StatusCode)"
- Ensure your URL string has no < or > anywhere.
Reference for how Prefect captures output:
- return_all=False → last stdout line as a string; no output → ""
- return_all=True → list of all stdout lines
prefect-shellCharlie Nash
12/15/2025, 10:36 PMimport requests, json, pyodbc, pandas as pd, re
from prefect import flow, task
from azure.storage.blob import BlobClient, BlobServiceClient
from azure.identity import DefaultAzureCredential
from prefect_snowflake.credentials import SnowflakeCredentials
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.backends import default_backend
from prefect_shell import shell_run_command
import snowflake.connector
import subprocess
import asyncio
#Get Raw Data from API
@flow (retries = 0, retry_delay_seconds = 20)
async def getData():
#Path to your PowerShell script
powershell_script_path = "DefenderGetAlerts.ps1"
out = await shell_run_command(command='pwsh -NoProfile -ExecutionPolicy Bypass -File ./DefenderGetAlerts.ps1', shell="bash", return_all=True)
print(out)
return out
#Load JSON data into Snowflake
@flow (retries = 0, retry_delay_seconds = 20, log_prints=True)
def loadJson(data):
print("data is below:")
print(data)
blob_service_client = BlobServiceClient(
account_url="<https://prefectscriptspy.blob.core.windows.net/>",
credential='mec3I4AojCiv764tjydLxY8twl16IwIe+9iJVsWgQnG2V+aAsrEIhhcUozcMMRxVIsQKz43V82Un+ASttIsz3w==')
container_name = "prefectscripts"
blob_name = "rsa_key.p8" # e.g., "private_key.pem"
container_client = blob_service_client.get_container_client(container_name)
blob_client = container_client.get_blob_client(blob_name)
try:
download_stream = blob_client.download_blob()
private_key_content = download_stream.readall()
# If the private key is text-based (e.g., PEM), you might want to decode it
# private_key_string = private_key_content.decode("utf-8")
print("Private key retrieved successfully.")
# Now, private_key_content holds the bytes of your private key
# You can process it further, e.g., load it into a cryptographic library
except Exception as e:
print(f"Error retrieving private key: {e}")
p_key = serialization.load_pem_private_key(
private_key_content,
password=None, # Set to your password if the key is encrypted
backend=default_backend()
)
USER = 'SVC_PF_USER'
ACCOUNT = 'xz30609.central-us.azure'
WAREHOUSE = 'COMPUTE_WH'
DATABASE = 'UNITEDCENTER_PROD_RAW_DB'
SCHEMA = 'MICROSOFT_SECURITY'
conn = snowflake.connector.connect(
user=USER,
private_key = p_key,
account=ACCOUNT,
warehouse=WAREHOUSE,
database=DATABASE,
schema=SCHEMA)
conn.timeout = 6000
conn.autocommit = True
dataToLoad = json.loads(data)
dataToLoad = json.dumps(dataToLoad)
# Call SP and trap Error if raised
cursor = conn.cursor()
# Execute Transactions load
cursor.execute('CALL load_defender_data(%s)', dataToLoad)
print('inserted data')
#except pyodbc.Error as err:
# print('Error !!!!! %s' % err)
#except:
# print('something else failed miserably')
conn.close()
print('closed db connection')
@flow (retries = 3, retry_delay_seconds = 20, log_prints=True)
def main_flow():
jsonData = asyncio.run(getData())
print(jsonData)
loadJson(jsonData)
if __name__ == '__main__':
jsonData = asyncio.run(getData())
print(type(jsonData))
loadJson(jsonData)Marvin
12/15/2025, 10:36 PMMarvin
12/15/2025, 10:37 PMprint(data) line inside loadJson() — it’s indented one level too deep and with a tab, while surrounding lines use spaces.
Problem area:
def loadJson(data):
print("data is below:")
print(data) # <-- extra indent + tab here
blob_service_client = BlobServiceClient(...
Fix:
def loadJson(data):
print("data is below:")
print(data)
blob_service_client = BlobServiceClient(...
Tip:
- Convert all indentation to spaces (4 spaces per level).
- To find stray tabs quickly: grep -nP '\t' your_file.py
- Python can also help: python -m tabnanny your_file.pyCharlie Nash
12/16/2025, 6:49 PMFileNotFoundError: [Errno 2] No such file or directory: 'powershell.exe'Marvin
12/16/2025, 6:49 PMMarvin
12/16/2025, 6:52 PMshell_run_command or ShellOperation), the default shell is:
- Windows: PowerShell
- Linux/macOS: bash
- On Windows hosts/containers without PowerShell (or with a missing PATH), calling PowerShell fails with FileNotFoundError: 'powershell.exe'.
How to fix it:
- If you intended to use bash/sh, explicitly set the shell in your task:
from prefect import flow
from prefect_shell import shell_run_command
@flow
async def run():
# Use bash on Unix-like systems
await shell_run_command("echo hello", shell="bash")
# Or sh
# await shell_run_command("echo hello", shell="sh")
- If you want PowerShell, install it in the runtime image/host or use PowerShell 7 (pwsh) if that’s what you have:
await shell_run_command("echo hello", shell="pwsh")
- If you’re running in Docker/Kubernetes, ensure the image includes the shell you’re targeting (PowerShell, bash, etc.) and that it’s on PATH.
Could you share:
- Where the flow is running (OS, container image, worker type)?
- Whether you’re using prefect-shell (and the code snippet)?
With that, I can give you a precise one-liner fix for your setup.Charlie Nash
12/16/2025, 6:59 PMCreateProcessCommon:798: execvpe(/bin/bash) failed: No such file or directorMarvin
12/16/2025, 6:59 PMMarvin
12/16/2025, 7:00 PMshell="bash" without Git Bash or WSL bash available.
How to fix:
- Install bash in your image/host:
- Debian/Ubuntu:
apt-get update && apt-get install -y bash
- Alpine:
apk add --no-cache bash
- RHEL/CentOS:
yum install -y bash
- Or use a shell that exists:
- On POSIX systems without bash:
from prefect_shell import shell_run_command
await shell_run_command("echo hello", shell="sh")
# On Alpine you can also use shell="ash"
- On Windows:
await shell_run_command("echo hello", shell="pwsh") # PowerShell 7
# or shell="powershell" if Windows PowerShell 5.x is installed
If you share where this is running (OS/container image/worker type) and how you’re calling the command, I can suggest the exact one-liner to fix it.Charlie Nash
12/16/2025, 7:03 PMRuntimeError: Command failed with exit code 127:
/tmp/prefect-4_uyidog: line 1: .\Get-Token-Intune.ps1: command not foundMarvin
12/16/2025, 7:03 PMMarvin
12/16/2025, 7:04 PM.\Get-Token-Intune.ps1 is a PowerShell path invocation that bash doesn’t understand.
What’s happening
- On non-Windows, Prefect’s shell helper defaults to bash unless you override it.
- Your command .\\Get-Token-Intune.ps1 is valid in PowerShell, not in bash.
- bash tries to execute a command literally named .\Get-Token-Intune.ps1, can’t find it, and exits with 127.
Fix options
- Run with PowerShell instead of bash:
from prefect import flow
from prefect_shell import shell_run_command
@flow
async def run():
# Windows PowerShell (5.x)
await shell_run_command(r".\Get-Token-Intune.ps1", shell="powershell", cwd="/path/to/script")
# PowerShell 7 (cross‑platform)
await shell_run_command("./Get-Token-Intune.ps1", shell="pwsh", cwd="/path/to/script")
Notes:
- Use shell="powershell" on Windows, shell="pwsh" if you have PowerShell 7 installed (Linux/macOS/Windows).
- Set cwd to the directory containing the script or use an absolute path.
- Or explicitly invoke PowerShell via the command:
await shell_run_command("pwsh -File ./Get-Token-Intune.ps1", cwd="/path/to/script")
# or on Windows:
await shell_run_command("powershell -File .\\Get-Token-Intune.ps1", cwd="C:\\path\\to\\script")
If you meant to use bash instead, you’d need a bash-compatible script and syntax (PowerShell .ps1 files won’t run under bash).
If you share:
- OS/runtime (Windows/Linux/macOS, container image),
- Whether you have PowerShell 7 (pwsh) or Windows PowerShell (powershell),
- The code snippet calling the command,
I can give you the precise one-liner that will work in your environment.Charlie Nash
12/16/2025, 7:24 PMMarvin
12/16/2025, 7:24 PMMarvin
12/16/2025, 7:26 PMfrom prefect import flow, task
@task(log_prints=True)
def t():
print("This will appear in Prefect logs")
@flow(log_prints=True)
def f():
print("Flow prints are logged too")
t()
f()
- Globally (affects all flows/tasks):
export PREFECT_LOGGING_LOG_PRINTS=True
- Notes:
- Only prints to stdout/stderr are captured.
- They are logged at INFO level.
- Requires a running flow/task context.
2) Recommended: use the run logger
- Prefer structured logging with get_run_logger() for reliability and control:
from prefect import flow, task
from prefect.logging import get_run_logger
@task
def t():
logger = get_run_logger()
<http://logger.info|logger.info>("Task message")
logger.warning("Heads up")
logger.debug("Debug details")
@flow
def f():
logger = get_run_logger()
<http://logger.info|logger.info>("Starting flow")
t()
- Benefits: proper context (flow/task IDs), levels, formatting, and automatic sending to the Prefect API.
Optional configuration you might find useful:
- Set default logging level:
export PREFECT_LOGGING_LEVEL=DEBUG
- Disable or tune API logging:
export PREFECT_LOGGING_TO_API_ENABLED=True
export PREFECT_LOGGING_TO_API_BATCH_INTERVAL=2.0
- Customize formatting/console color/markup via other PREFECT_LOGGING_* settings if needed.
If you share how you’re running (local, Docker, worker type) I can suggest the minimal settings to get your prints into the UI with your setup.Charlie Nash
12/16/2025, 7:29 PMTask run 'shell_run_command-406' - Task run failed with exception: FileNotFoundError(2, 'The system cannot find the file specified', None, 2, None)Marvin
12/16/2025, 7:30 PMMarvin
12/16/2025, 7:31 PMprefect-shell this usually comes from one of these:
- The shell executable is missing
- Windows default is PowerShell. If powershell.exe or pwsh.exe isn’t installed or on PATH, process creation fails with FileNotFoundError.
- On Windows, setting shell="bash" will also fail unless you actually have bash installed.
- The working directory isn’t what you expect
- Your files may be “in the same directory” as your code locally, but the flow run’s working directory can differ (e.g., in a worker, container, or deployment). Then relative paths won’t resolve.
- The command references a file/executable that doesn’t exist at the given path in the run environment.
What to try
1) Verify the shell exists and matches your environment
- Windows PowerShell:
await shell_run_command("echo hello", shell="powershell")
- PowerShell 7:
await shell_run_command("echo hello", shell="pwsh")
- Linux/macOS:
await shell_run_command("echo hello", shell="bash") # or shell="sh"
If this simple echo fails, the chosen shell isn’t available.
2) Set the working directory explicitly or use absolute paths
- Point cwd to the folder that contains your files:
await shell_run_command("./my_script.ps1 arg1", shell="pwsh", cwd="C:\\path\\to\\dir")
# or
await shell_run_command("./my_script.sh arg1", shell="bash", cwd="/app/dir")
3) Match command syntax to the shell you’re using
- PowerShell (Windows PowerShell or pwsh):
await shell_run_command("./Get-Token-Intune.ps1", shell="pwsh", cwd="C:\\path\\to\\dir")
- Bash/sh:
await shell_run_command("./script.sh", shell="bash", cwd="/path/to/dir")
# or: await shell_run_command("bash script.sh", shell="bash", cwd="/path/to/dir")
If you can share:
- OS/runtime (Windows/Linux/macOS, container image),
- Your shell_run_command call (including shell= and cwd=),
- The exact command you’re running,
I can pinpoint the cause and give you the exact one-liner fix.Charlie Nash
12/16/2025, 7:36 PMRuntimeError: Command failed with exit code 2:
./DefenderGetAlerts.ps1: line 3: =: command not found
./DefenderGetAlerts.ps1: line 6: syntax error near unexpected token `('
./DefenderGetAlerts.ps1: line 6: `$dateTime = (Get-Date).ToUniversalTime().AddHours(-48).ToString("o")'
The code block is:
# Returns Alerts created in the past 48 hours.
$token = ./Get-Token.ps1 #run the script Get-Token.ps1 - make sure you are running this script from the same folder of Get-Token.ps1
# Get Alert from the last 48 hours. Make sure you have alerts in that time frame.
$dateTime = (Get-Date).ToUniversalTime().AddHours(-48).ToString("o")
# The URL contains the type of query and the time filter we create above
# Read more about [other query options and filters](get-alerts.md).
$url = "<https://api.securitycenter.microsoft.com/api/alerts>?`$filter=alertCreationTime ge $dateTime"
# Set the WebRequest headers
$headers = @{
'Content-Type' = 'application/json'
Accept = 'application/json'
Authorization = "Bearer $token"
}
# Send the webrequest and get the results.
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop
# Extract the alerts from the results.
$alerts = ($response | ConvertFrom-Json).value | ConvertTo-Json
# Get string with the execution time. We concatenate that string to the output file to avoid overwrite the file
$dateTimeForFileName = Get-Date -Format o | foreach {$_ -replace ":", "."}
Write-Output $alertsMarvin
12/16/2025, 7:36 PMMarvin
12/16/2025, 7:38 PMpwsh) explicitly and have it installed.
How to fix
- Run the script with PowerShell, not bash, and set the working directory to where the files are:
- Linux/macOS (requires PowerShell 7 installed):
from prefect import flow
from prefect_shell import shell_run_command
@flow
async def run():
await shell_run_command(
"./DefenderGetAlerts.ps1",
shell="pwsh",
cwd="/path/to/dir"
)
- Windows:
from prefect import flow
from prefect_shell import shell_run_command
@flow
async def run():
await shell_run_command(
".\\DefenderGetAlerts.ps1",
shell="powershell",
cwd="C:\\path\\to\\dir"
)
- Or call PowerShell explicitly from the default bash shell:
await shell_run_command("pwsh -File ./DefenderGetAlerts.ps1", cwd="/path/to/dir")
Make your script resilient to the working directory
- Inside your .ps1, reference the sibling script via the script’s own directory:
$token = & (Join-Path $PSScriptRoot 'Get-Token.ps1')
- Build the URL cleanly to avoid escaping confusion:
$base = '<https://api.securitycenter.microsoft.com/api/alerts>'
$url = "$base?`$filter=alertCreationTime ge $dateTime"
If this still fails, please share:
- OS/container image where the flow runs, and whether `pwsh`/`powershell` is installed
- Your exact shell_run_command(...) call (including shell= and cwd=)
- The path to the scripts at runtime
I’ll tailor a one-liner that works in your environment.Charlie Nash
12/16/2025, 7:54 PMFileNotFoundError: [WinError 2] The system cannot find the file specifiedMarvin
12/16/2025, 7:54 PMMarvin
12/16/2025, 7:55 PMawait shell_run_command("echo hello", shell="powershell")
# or if you have PowerShell 7:
await shell_run_command("echo hello", shell="pwsh")
- If even echo fails, the chosen shell isn’t available on PATH.
- Wrong working directory (relative paths don’t resolve)
- The flow’s runtime CWD in a worker/container may differ from your local dev folder.
- Set cwd explicitly or use absolute paths:
await shell_run_command(".\\DefenderGetAlerts.ps1", shell="powershell", cwd="C:\\path\\to\\dir")
# or
await shell_run_command("pwsh -File ./DefenderGetAlerts.ps1", cwd="C:\\path\\to\\dir")
- File not included in the run environment
- If this is a deployment, only files available in the runtime (e.g., your repo or image) are present. Local uncommitted files won’t be there. Ensure scripts/data are checked into the repo or baked into the image and pulled at run time.
- Path/quoting issues on Windows
- Paths with spaces must be quoted if you build commands yourself.
- Prefer passing paths via cwd or use absolute paths.
Quick debugging steps:
- Print the current working directory and list files to confirm:
await shell_run_command("Get-Location; Get-ChildItem", shell="powershell")
- Check a file exists before running:
from pathlib import Path
p = Path("DefenderGetAlerts.ps1")
assert p.exists(), f"Missing: {p.resolve()}"
If you share:
- How you’re invoking the command (your shell_run_command(...) call),
- The OS/container image and whether you have powershell or pwsh,
- Whether this is a local run or a deployment/worker,
I’ll pinpoint the cause and give you the exact one-liner fix.Charlie Nash
12/16/2025, 9:27 PM<https://trufflesecurity.com/trufflehog>Marvin
12/16/2025, 9:27 PMMarvin
12/16/2025, 9:28 PMprefect-shell (e.g., shell_run_command("trufflehog ...")) and capture the results. Happy to sketch a quick example based on your environment.Charlie Nash
12/16/2025, 10:25 PMOSError: [WinError 193] %1 is not a valid Win32 applicationMarvin
12/16/2025, 10:25 PMMarvin
12/16/2025, 10:26 PMshell="powershell" (Windows) or shell="pwsh" (PowerShell 7 on any OS)
- .bat/.cmd should be run by PowerShell (works) or via cmd /c
- .py needs python script.py (Windows doesn’t honor shebangs)
- Trying to run a non-Windows binary (e.g., Linux ELF) on Windows
- Passing a path that’s a folder or wrong file
- Shebang-only scripts (no extension) on Windows without an association
How to fix (pick what fits your case):
- PowerShell script
from prefect import flow
from prefect_shell import shell_run_command
@flow
async def run():
await shell_run_command(".\\script.ps1", shell="powershell", cwd="C:\\path\\to\\dir")
# or, with PowerShell 7 installed:
# await shell_run_command("./script.ps1", shell="pwsh", cwd="C:\\path\\to\\dir")
- Batch script
await shell_run_command(".\\script.bat", shell="powershell", cwd="C:\\path\\to\\dir")
# or
await shell_run_command("cmd /c script.bat", shell="powershell", cwd="C:\\path\\to\\dir")
- Python script
await shell_run_command('python "C:\\path\\to\\script.py"', shell="powershell")
- Executable: ensure it’s a native Windows .exe and the path is correct
await shell_run_command('"C:\\path\\to\\tool.exe" --help', shell="powershell")
Quick diagnostics:
- Confirm the working directory and files:
await shell_run_command("Get-Location; Get-ChildItem", shell="powershell")
- Verify interpreter availability:
await shell_run_command("Get-Command powershell; Get-Command pwsh", shell="powershell")
- Check the target path in Python before running:
from pathlib import Path
p = Path("C:\\path\\to\\thing")
print(p.resolve(), p.exists(), p.is_dir())
If you share:
- The exact shell_run_command(...) call (command, shell=, cwd=),
- What file you’re trying to run,
- Where the flow runs (local vs worker, container image, OS),
I’ll pinpoint the cause and give you the exact one-liner fix.Charlie Nash
12/16/2025, 10:51 PMTask run 'shell_run_command-677' - Task run failed with exception: FileNotFoundError(2, 'The system cannot find the file specified', None, 2, None)Marvin
12/16/2025, 10:51 PMMarvin
12/16/2025, 10:51 PMpowershell.exe (or pwsh) isn’t installed/on PATH, process creation fails.
- Wrong working directory: The flow’s runtime CWD (in a worker/container/deployment) may not be the folder you expect, so relative paths don’t resolve.
- File not present in the runtime: The script/exe isn’t included in the image/repo that the deployment runs.
- Path/quoting issues: Windows paths with spaces or backslashes can break if not quoted/escaped.
How to fix:
- Verify and choose the right shell:
await shell_run_command("echo hello", shell="powershell")
# or if you have PowerShell 7:
await shell_run_command("echo hello", shell="pwsh")
If this fails, that shell isn’t available on PATH.
- Set the working directory or use absolute paths:
await shell_run_command(".\\DefenderGetAlerts.ps1", shell="powershell", cwd="C:\\path\\to\\dir")
# or
await shell_run_command("pwsh -File ./DefenderGetAlerts.ps1", cwd="C:\\path\\to\\dir")
- Match the command to the shell:
- PowerShell script: shell="powershell" (Windows) or shell="pwsh" (PowerShell 7)
- Batch: cmd /c script.bat (or run from PowerShell)
- Python: python path\\to\\script.py (Windows doesn’t use shebangs)
Quick diagnostics:
# Check CWD and files (PowerShell)
await shell_run_command("Get-Location; Get-ChildItem", shell="powershell")
# Confirm your target exists before running (in Python)
from pathlib import Path
p = Path("C:\\path\\to\\your_file.ps1")
print(p.resolve(), p.exists(), p.is_file())
If you share:
- Your exact shell_run_command(...) call (command, shell=, cwd=),
- Where the flow runs (local vs worker, Docker image),
- The file you’re trying to run,
I can pinpoint the cause and give you the precise one-liner fix.Charlie Nash
12/16/2025, 10:58 PMout = await shell_run_command(command=f'".\\DefenderGetAlerts.ps1"', shell="pwsh", return_all=False)Marvin
12/16/2025, 10:58 PMMarvin
12/16/2025, 10:59 PMshell="pwsh" which is PowerShell 7. On many Windows hosts only Windows PowerShell 5.x (powershell.exe) is installed.
- If pwsh is missing, Prefect can’t start the process and you’ll get FileNotFoundError(2).
- Fix: either install PowerShell 7, or switch to Windows PowerShell:
out = await shell_run_command(
command=".\\DefenderGetAlerts.ps1",
shell="powershell", # use this if pwsh is not available
return_all=False
)
To confirm availability:
await shell_run_command("Get-Command pwsh", shell="powershell")
2) The command string is quoted, which prevents execution
- You passed command=f'".\\DefenderGetAlerts.ps1"' (note the surrounding quotes). In PowerShell, a quoted path is just a string literal, not an invocation.
- Use either the bare path or the call operator & when quoting:
- Bare (no spaces in path):
out = await shell_run_command(
command=".\\DefenderGetAlerts.ps1",
shell="pwsh",
return_all=False
)
- With quoting or spaces in the path:
out = await shell_run_command(
command='& ".\\DefenderGetAlerts.ps1"',
shell="pwsh",
return_all=False
)
Also make sure the working directory is correct
- .\DefenderGetAlerts.ps1 is relative to the process CWD, which may not be your project folder in a worker/deployment.
- Set cwd to the folder containing the script or use an absolute path:
out = await shell_run_command(
command='& ".\\DefenderGetAlerts.ps1"',
shell="pwsh",
cwd="C:\\path\\to\\dir",
return_all=False
)
Quick diagnostics
- Check current directory and files:
await shell_run_command("Get-Location; Get-ChildItem", shell="powershell")
- Verify the script exists from Python before calling:
from pathlib import Path
p = Path("C:\\path\\to\\dir\\DefenderGetAlerts.ps1")
print(p.exists(), p.is_file())
If you confirm whether pwsh is available and where the script lives (and whether this is local or a deployment/worker), I can give you the exact one-liner that will work in your environment.