https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> In Prefect 3.7.0 I use an automations...
# ask-marvin
r
@Marvin In Prefect 3.7.0 I use an automations.yaml to define automations which are then created using a bash script in the cicd plan that calls prefect automations create --from-file automations.yaml Aside from the tricky part to get the id of the notifications Block, that works fine. However, if the automation already exists it creates another one. So over time there are lots of duplicates. Ideally there would be a prefect automations upsert --from-file cli function, but there isn't. Can you help to write a bash script that checks for already existing automations, and if any exist the script extracts the id and deletes it. Then it calls the prefect automations create cli command thereby ensuring no duplicates. Basically I'm looking for a workaround for an upsert but don't want to write and debug it myself, but get you to do it for me.
m
✅ thought for 114.1 seconds
Yes — a practical workaround is: 1. Read the automation names from your 
automations.yaml
2. Query existing automations from Prefect 3. Delete any existing automation(s) with matching names 4. Re-run 
prefect automation create --from-file automations.yaml
A few important notes first: * In Prefect 3.x, there is no native upsert command for automations. * 
prefect automation delete
can delete by name, but if duplicates already exist, deleting by name may fail because the name is not unique. * So the safest approach is to: * list existing automations as JSON * find matching names * delete by ID I can give you a bash script for that. Assumptions * You have 
jq
installed * Your Prefect CLI is authenticated against the right workspace * Your file contains one or more automations with a top-level 
automations:
array, or a single automation with a 
name:
field Here’s a robust script:
Copy code
bash
#!/usr/bin/env bash
set -euo pipefail

AUTOMATIONS_FILE="${1:-automations.yaml}"

if [[ ! -f "$AUTOMATIONS_FILE" ]]; then
  echo "Error: file not found: $AUTOMATIONS_FILE" >&2
  exit 1
fi

command -v prefect >/dev/null 2>&1 || {
  echo "Error: prefect CLI not found" >&2
  exit 1
}

command -v jq >/dev/null 2>&1 || {
  echo "Error: jq is required but not installed" >&2
  exit 1
}

command -v python >/dev/null 2>&1 || command -v python3 >/dev/null 2>&1 || {
  echo "Error: python or python3 is required" >&2
  exit 1
}

PYTHON_BIN="$(command -v python || command -v python3)"

echo "Using automations file: $AUTOMATIONS_FILE"

# Extract automation names from YAML or JSON using Python
AUTOMATION_NAMES_JSON="$("$PYTHON_BIN" - <<'PY' "$AUTOMATIONS_FILE"
import json
import sys
from pathlib import Path

path = Path(sys.argv[1])
text = path.read_text()

data = None

if path.suffix.lower() == ".json":
    data = json.loads(text)
else:
    try:
        import yaml
    except ImportError:
        print("ERROR: PyYAML is required to parse YAML files", file=sys.stderr)
        sys.exit(2)
    data = yaml.safe_load(text)

names = []

if isinstance(data, dict) and "automations" in data and isinstance(data["automations"], list):
    for item in data["automations"]:
        if isinstance(item, dict) and item.get("name"):
            names.append(item["name"])
elif isinstance(data, list):
    for item in data:
        if isinstance(item, dict) and item.get("name"):
            names.append(item["name"])
elif isinstance(data, dict) and data.get("name"):
    names.append(data["name"])

print(json.dumps(names))
PY
)"

if [[ "$AUTOMATION_NAMES_JSON" == "" || "$AUTOMATION_NAMES_JSON" == "[]" ]]; then
  echo "No automation names found in $AUTOMATIONS_FILE" >&2
  exit 1
fi

echo "Automation names found in file:"
echo "$AUTOMATION_NAMES_JSON" | jq -r '.[] | " - \(.)"'

echo
echo "Fetching existing automations from Prefect..."

EXISTING_AUTOMATIONS_JSON="$(prefect automation ls --output json)"

MATCHING_IDS="$(
  jq -r \
    --argjson names "$AUTOMATION_NAMES_JSON" \
    '
    .[]
    | select(.name as $n | $names | index($n))
    | .id
    ' <<< "$EXISTING_AUTOMATIONS_JSON"
)"

MATCHING_ROWS="$(
  jq -r \
    --argjson names "$AUTOMATION_NAMES_JSON" \
    '
    .[]
    | select(.name as $n | $names | index($n))
    | "\(.name)\t\(.id)"
    ' <<< "$EXISTING_AUTOMATIONS_JSON"
)"

if [[ -n "$MATCHING_ROWS" ]]; then
  echo "Existing matching automations found. Deleting them first:"
  while IFS=$'\t' read -r name id; do
    [[ -z "$id" ]] && continue
    echo " - Deleting automation: $name ($id)"
    prefect automation delete --id "$id"
  done <<< "$MATCHING_ROWS"
else
  echo "No matching existing automations found."
fi

echo
echo "Creating automations from file..."
prefect automation create --from-file "$AUTOMATIONS_FILE"

echo
echo "Done. Existing automations with matching names were removed, and the file was re-created."
How to use it
Copy code
bash
./upsert-automations.sh automations.yaml
If your CI runner only has 
python3
, that’s fine — the script handles that. Requirements If your YAML file is actually YAML, the script uses 
PyYAML
. If that’s missing, install it in CI:
Copy code
bash
pip install pyyaml
If you want, you can also add 
jq
installation depending on your runner image. Why this works better than delete-by-name If duplicates already exist, 
prefect automation delete "my-name"
may not be enough because the CLI can require a unique match. Deleting by 
id
avoids ambiguity. One caveat This is really a delete and recreate workaround, not a true update/upsert. That means: * automation IDs will change on each run * if you were relying on stable automation IDs elsewhere, this could matter If you'd like, I can also give you a more advanced version that: * compares names * if exactly one match exists, updates it instead of deleting/recreating * if multiple duplicates exist, deletes all and recreates once That would behave more like a true upsert.
r
@Marvin It's not yet robust. Running that script in debian bullseye actually crashes. Yes, jq is installed. I noticed shellcheck gave several warnings about the script with regards to not working with POSIX. Can you improve the script to make it even more robust and work in bullseye?
m
✅ thought for 93.2 seconds
Yes — and there’s an important correction to make first: 
prefect automation ls --output json
is not supported in Prefect 3.x. That means my earlier script was relying on a CLI flag that doesn’t exist, so a crash on Bullseye is not surprising. Thanks for pushing on robustness. If you want something that works reliably on Debian Bullseye, the best approach is: * use a POSIX 
sh
 script, not Bash-specific features * use Python to: * parse 
automations.yaml
* query existing automations via the Prefect client API * use the Prefect CLI only for the actual 
delete
and 
create
That avoids brittle parsing of the table output from 
prefect automation ls
. Here’s a much more robust Bullseye-friendly version.
Copy code
sh
#!/bin/sh
set -eu

# Usage:
#   ./prefect-automation-recreate.sh automations.yaml
#
# Requirements:
#   - prefect CLI
#   - python3
#   - jq
#   - PyYAML (if automations file is YAML)
#
# Notes:
#   - POSIX sh compatible
#   - Deletes existing automations with matching names, by ID
#   - Recreates from the supplied file
#   - Avoids parsing `prefect automation ls` table output

FILE=${1:-automations.yaml}

log() {
    printf '%s\n' "$*" >&2
}

die() {
    log "ERROR: $*"
    exit 1
}

need_cmd() {
    command -v "$1" >/dev/null 2>&1 || die "Required command not found: $1"
}

need_cmd prefect
need_cmd python3
need_cmd jq

[ -f "$FILE" ] || die "File not found: $FILE"

TMP_NAMES=$(mktemp)
TMP_AUTOS=$(mktemp)
TMP_MATCHES=$(mktemp)

cleanup() {
    rm -f "$TMP_NAMES" "$TMP_AUTOS" "$TMP_MATCHES"
}
trap cleanup EXIT HUP INT TERM

log "Reading automation names from $FILE ..."

python3 - "$FILE" >"$TMP_NAMES" <<'PY'
import json
import sys
from pathlib import Path

path = Path(sys.argv[1])
text = path.read_text(encoding="utf-8")

if path.suffix.lower() == ".json":
    data = json.loads(text)
else:
    try:
        import yaml
    except ImportError:
        print("PyYAML is required for YAML files", file=sys.stderr)
        sys.exit(2)
    data = yaml.safe_load(text)

names = []

if isinstance(data, dict) and isinstance(data.get("automations"), list):
    for item in data["automations"]:
        if isinstance(item, dict) and item.get("name"):
            names.append(item["name"])
elif isinstance(data, list):
    for item in data:
        if isinstance(item, dict) and item.get("name"):
            names.append(item["name"])
elif isinstance(data, dict) and data.get("name"):
    names.append(data["name"])

print(json.dumps(names))
PY

if [ ! -s "$TMP_NAMES" ]; then
    die "Could not extract automation names from $FILE"
fi

if [ "$(jq 'length' "$TMP_NAMES")" -eq 0 ]; then
    die "No automation names found in $FILE"
fi

log "Automation names found:"
jq -r '.[] | " - " + .' "$TMP_NAMES" >&2

log "Fetching existing automations from Prefect API ..."

python3 >"$TMP_AUTOS" <<'PY'
import asyncio
import json
import sys

from prefect.client.orchestration import get_client

async def main():
    async with get_client() as client:
        automations = await client.read_automations()
        result = [json.loads(a.model_dump_json()) for a in automations]
        print(json.dumps(result))

asyncio.run(main())
PY

if [ ! -s "$TMP_AUTOS" ]; then
    die "Failed to fetch existing automations"
fi

jq \
  --slurpfile names "$TMP_NAMES" \
  '
  .[]
  | select(.name as $n | $names[0] | index($n))
  | {id, name}
  ' "$TMP_AUTOS" >"$TMP_MATCHES"

if [ -s "$TMP_MATCHES" ] && [ "$(jq -s 'length' "$TMP_MATCHES")" -gt 0 ]; then
    log "Existing matching automations found; deleting them first ..."
    jq -r '[.name, .id] | @tsv' "$TMP_MATCHES" |
    while IFS="$(printf '\t')" read -r name id; do
        [ -n "$id" ] || continue
        log "Deleting: $name ($id)"
        PREFECT_CLI_PROMPT=false prefect automation delete --id "$id"
    done
else
    log "No matching existing automations found."
fi

log "Creating automations from $FILE ..."
prefect automation create --from-file "$FILE"

log "Done."
Why this version is more robust
* No Bash arrays * No 
[[ ... ]]
* No 
pipefail
dependency * No parsing of Rich table output from 
prefect automation ls
* Uses 
mktemp
+ 
trap
for clean temp file handling * Uses the Prefect Python client to get automations in structured form * Deletes by 
id
, which handles existing duplicates correctly Why Bullseye should like this better Debian Bullseye is happier with plain 
/bin/sh
scripts than scripts that accidentally depend on Bash features. ShellCheck warnings about POSIX compatibility were a good sign that the earlier version should be simplified. Dependencies to ensure in CI
Copy code
sh
apt-get update
apt-get install -y python3 jq
pip install prefect pyyaml
If Prefect is already installed in your CI image, you may only need:
Copy code
sh
pip install pyyaml
One more important note I used:
Copy code
sh
PREFECT_CLI_PROMPT=false prefect automation delete --id "$id"
If your environment still prompts, switch that line to:
Copy code
sh
yes | prefect automation delete --id "$id"
But I’d try the first version first, since piping 
yes
is uglier and can mask problems. Even better option: true update-like behavior If you want, I can give you a second version that is even better than delete/recreate: * if exactly one automation with that name exists: 
prefect automation update --id ... --from-file ...
* if more than one exists: delete all duplicates, then create once * if none exists: create That behaves much closer to a real upsert and avoids changing IDs unnecessarily when there’s only one existing automation.
r
@Marvin Ok, I think we're finally getting somewhere. So, let's finish it off properly. Can you give the even better option that is true update-like behaviour? But still keep the bulleye bits from your latest script.
m
✅ thought for 28.6 seconds
Absolutely — that’s the better design. For a true update-like workflow on Debian Bullseye, the safest behavior is: * parse the automations from the file * for each automation name: * 0 matches → create it * 1 match → update it in place using its existing ID *
1 matches
 → delete all duplicates, then create one fresh copy That preserves stable IDs when things are already clean, and self-heals old duplication if your workspace has accumulated junk. Because 
prefect automation update
operates on one automation at a time, the cleanest approach is to process each automation individually, not pass the whole file to one command. Below is a POSIX 
sh
 Bullseye-friendly script that does that. ```sh #!/bin/sh set -eu # Prefect automation upsert-like script with true update behavior # # Behavior per automation name: # - 0 existing matches -> create # - 1 existing match -> update by id # - >1 existing matches -> delete all matches, then create once # # Requirements: # - /bin/sh # - prefect CLI # - python3 # - jq # - PyYAML for YAML input # # Usage: # ./prefect-automation-upsert.sh automations.yaml FILE=${1:-automations.yaml} log() { printf '%s\n' "$*" >&2 } die() { log "ERROR: $*" exit 1 } need_cmd() { command -v "$1" >/dev/null 2>&1 || die "Required command not found: $1" } need_cmd prefect need_cmd python3 need_cmd jq [ -f "$FILE" ] || die "File not found: $FILE" TMP_DIR=$(mktemp -d) cleanup() { rm -rf "$TMP_DIR" } trap cleanup EXIT HUP INT TERM AUTOS_JSON="$TMP_DIR/automations.json" EXISTING_JSON="$TMP_DIR/existing.json" ITEMS_NDJSON="$TMP_DIR/items.ndjson" log "Parsing automations from $FILE ..." python3 - "$FILE" >"$AUTOS_JSON" <<'PY' import json import sys from pathlib import Path path = Path(sys.argv[1]) text = path.read_text(encoding="utf-8") if path.suffix.lower() == ".json": data = json.loads(text) else: try: import yaml except ImportError: print("PyYAML is required for YAML files", file=sys.stderr) sys.exit(2) data = yaml.safe_load(text) items = [] if isinstance(data, dict) and isinstance(data.get("automations"), list): for item in data["automations"]: if isinstance(item, dict) and item.get("name"): items.append(item) elif isinstance(data, list): for item in data: if isinstance(item, dict) and item.get("name"): items.append(item) elif isinstance(data, dict) and data.get("name"): items.append(data) print(json.dumps(items)) PY if [ ! -s "$AUTOS_JSON" ]; then die "Failed to parse automations from $FILE" fi COUNT=$(jq 'length' "$AUTOS_JSON") [ "$COUNT" -gt 0 ] || die "No automations found in $FILE" log "Found $COUNT automation(s) in file." log "Fetching existing automations from Prefect API ..." python3 >"$EXISTING_JSON" <<'PY' import asyncio import json from prefect.client.orchestration import get_client async def main(): async with get_client() as client: automations = await client.read_automations() result = [json.loads(a.model_dump_json()) for a in automations] print(json.dumps(result)) asyncio.run(main()) PY [ -s "$EXISTING_JSON" ] || die "Failed to fetch existing automations from Prefect" jq -c '.[]' "$AUTOS_JSON" >"$ITEMS_NDJSON" log "Processing automations ..." while IFS= read -r ITEM_JSON do [ -n "$ITEM_JSON" ] || continue NAME=$(printf '%s\n' "$ITEM_JSON" | jq -r '.name') [ -n "$NAME" ] || die "Encountered automation without a name" SAFE_NAME=$(printf '%s' "$NAME" | tr '/[space]' '____') ITEM_FILE="$TMP_DIR/$SAFE_NAME.json" MATCHES_FILE="$TMP_DIR/$SAFE_NAME.matches.json" printf '%s\n' "$ITEM_JSON" >"$ITEM_FILE" jq \ --arg name "$NAME" \ '[ .[] | select(.name == $name) | {id, name} ]' \ "$EXISTING_JSON" >"$MATCHES_FILE" MATCH_COUNT=$(jq 'length' "$MATCHES_FILE") if [ "$MATCH_COUNT" -eq 0 ]; then log "CREATE: $NAME"
prefect automation create --from-file "$ITEM_FILE" continue fi if [ "$MATCH_COUNT" -eq 1 ]; then ID=$(jq -r '.[0].id' "$MATCHES_FILE") [ -n "$ID" ] || die "Could not determine ID for automation: $NAME" log "UPDATE: $NAME ($ID)" prefect automation update --id "$ID" --from-file "$ITEM_FILE" continue fi log "DEDUPE: $NAME has $MATCH_COUNT existing entries; deleting all and recreating once" jq -r '.[].id' "$MATCHES_FILE" | while IFS= read -r ID do [ -n "$ID" ] || continue log "Deleting duplicate: $NAME ($ID)" PREFECT_CLI_PROMPT=false prefect automation delete --id "$ID" done log "RECREATE: $NAME" prefect automation create --from-file "$ITEM_FILE" done <"$ITEMS_NDJSON" log "Done."
Copy code
*Why this is the better version*

Compared with delete-and-recreate for everything, this one is better because:

* *clean existing automation* → updated in place
* *duplicate mess* → repaired automatically
* *new automation* → created normally

So it behaves much more like a real upsert.

*Supported file shapes*

This script handles:

1. A single automation object:
```yaml
name: my-automation
trigger:
  ...
actions:
  ...
2. A list of automations:
Copy code
yaml
- name: automation-one
  trigger:
    ...
  actions:
    ...
- name: automation-two
  trigger:
    ...
  actions:
    ...
3. A top-level 
automations:
wrapper:
Copy code
yaml
automations:
  - name: automation-one
    trigger:
      ...
    actions:
      ...
  - name: automation-two
    trigger:
      ...
    actions:
      ...
Bullseye-friendly choices kept from the prior script * POSIX 
sh
* no Bash arrays * no 
[[ ... ]]
* no process substitution * no 
pipefail
* uses 
mktemp -d
* uses 
trap
cleanup * uses Python instead of trying to scrape CLI table output Dependencies
Copy code
sh
apt-get update
apt-get install -y python3 jq
pip install prefect pyyaml
One subtle point This script fetches the existing automations once at the beginning. That is usually what you want in CI. If your file itself contains duplicate names, then the behavior becomes: * first item may create/update * second item with same name may act against the original initial snapshot, not the just-modified state So if duplicate names inside the same file are possible, I should harden this one more by: * validating the input file for duplicate names first * failing fast if duplicates are present That would be my recommendation. If you want, I can give you a final hardened version with: * duplicate-name validation inside the file * clearer exit codes * 
--dry-run
* optional 
--delete-extra
/ 
--strict
modes That would be the version I’d actually put into CI.
r
@Marvin Ok, that looks good. Thank you! Please celebrate by listening to Ave Maria as sung by Pavarotti. Everyone likes Ave Maria and especially when it's sung by Pavarotti.
m
✅ thought for 7.4 seconds
Glad we got it into good shape 🙂 I can’t actually listen to music, but that does sound like a strong victory-lap choice. If you want the proper celebration track, here you go: Pavarotti - Ave Maria And nice catch on pushing for Bullseye-safe and actually robust behavior — that was the right instinct.
r
@Marvin Your most recent script above errors out. For example you use --from-file rather than --from-json, but the files you create are json. However, even after changing your script to use --from-json it still errors out as the json files have some validation issues. Also your script has python program embedded in it. That would be better as a separate .py file and then called by the bash script. Can you rewrite the script to use yaml instead of json? And also to keep the pure python part as separate python files to be called. You can assume yq is installed, as well as uv.
m
✅ thought for 109.5 seconds
Yes — good catch again. You were right on both points: * I was generating per-item JSON files but then using 
--from-file
as if that were guaranteed to be fine in the way I structured them. * Separating the Python helper logic into standalone files is cleaner and much easier to debug in CI. Given your constraints, the best version is: * shell script stays POSIX/Bullseye-friendly * Python helpers are separate 
.py
files * per-automation temp files are YAML, not JSON * 
yq
is used to split the source file into one YAML file per automation * Python helper queries existing automations from the Prefect API and returns name/id matches * shell script does: * 0 matches -> create * 1 match -> update * >1 matches -> delete all, then create That’s the most maintainable version. Below is a working structure. --- Files to create 1. 
prefect-automation-upsert.sh
2. 
list_automations.py
--- 1) 
prefect-automation-upsert.sh
 ```sh #!/bin/sh set -eu # Usage: # ./prefect-automation-upsert.sh automations.yaml # # Requirements: # - /bin/sh # - prefect CLI # - jq # - yq # - uv # # Python deps used via uv: # - prefect # # Behavior per automation name: # - 0 matches -> create # - 1 match -> update # - >1 match -> delete all matches, then create once FILE=${1:-automations.yaml} SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) LIST_SCRIPT="$SCRIPT_DIR/list_automations.py" log() { printf '%s\n' "$*" >&2 } die() { log "ERROR: $*" exit 1 } need_cmd() { command -v "$1" >/dev/null 2>&1 || die "Required command not found: $1" } need_cmd prefect need_cmd jq need_cmd yq need_cmd uv [ -f "$FILE" ] || die "File not found: $FILE" [ -f "$LIST_SCRIPT" ] || die "Python helper not found: $LIST_SCRIPT" TMP_DIR=$(mktemp -d) cleanup() { rm -rf "$TMP_DIR" } trap cleanup EXIT HUP INT TERM ITEMS_DIR="$TMP_DIR/items" mkdir -p "$ITEMS_DIR" EXISTING_JSON="$TMP_DIR/existing.json" INPUT_NAMES="$TMP_DIR/input-names.txt" INPUT_NAMES_SORTED="$TMP_DIR/input-names-sorted.txt" DUP_NAMES="$TMP_DIR/duplicate-names.txt" log "Validating and splitting $FILE into per-automation YAML files ..." # Detect input shape and split into one YAML file per automation. # Each output file contains exactly one automation object. count=$(yq eval ' if type == "!!map" and has("automations") then .automations | length elif type == "!!seq" then length elif type == "!!map" and has("name") then 1 else 0 end ' "$FILE") [ "$count" -gt 0 ] || die "No automations found in $FILE" i=0 while [ "$i" -lt "$count" ] do out="$ITEMS_DIR/automation-$i.yaml" yq eval " if type == \"!!map\" and has(\"automations\") then .automations[$i] elif type == \"!!seq\" then .[$i] else . end " "$FILE" > "$out" name=$(yq eval -r '.name' "$out") [ "$name" != "null" ] || die "Automation at index $i has no name" printf '%s\n' "$name" >> "$INPUT_NAMES" i=$((i + 1)) done sort "$INPUT_NAMES" > "$INPUT_NAMES_SORTED" uniq -d "$INPUT_NAMES_SORTED" > "$DUP_NAMES" || true if [ -s "$DUP_NAMES" ]; then log "Duplicate automation names found in input file:" sed 's/^/ - /' "$DUP_NAMES" >&2 die "Input file must not contain duplicate automation names" fi log "Found $count automation(s) in input." log "Fetching existing automations from Prefect API ..." uv run "$LIST_SCRIPT" > "$EXISTING_JSON" [ -s "$EXISTING_JSON" ] || die "Failed to fetch existing automations" # Validate helper output early jq -e 'type == "array"' "$EXISTING_JSON" >/dev/null 2>&1 || \ die "Python helper did not return a JSON array" i=0 while [ "$i" -lt "$count" ] do item_file="$ITEMS_DIR/automation-$i.yaml" name=$(yq eval -r '.name' "$item_file") log "Processing: $name" matches_file="$TMP_DIR/matches-$i.json" jq --arg name "$name" '[ .[] | select(.name == $name) ]' "$EXISTING_JSON" > "$matches_file"
match_count=$(jq 'length' "$matches_file") if [ "$match_count" -eq 0 ]; then log "CREATE: $name" prefect automation create --from-file "$item_file" elif [ "$match_count" -eq 1 ]; then id=$(jq -r '.[0].id' "$matches_file") [ "$id" != "null" ] || die "Could not determine id for $name" log "UPDATE: $name ($id)" prefect automation update --id "$id" --from-file "$item_file" else log "DEDUPE: $name has $match_count matches; deleting all and recreating once" jq -r '.[].id' "$matches_file" | while IFS= read -r id do [ -n "$id" ] || continue log "Deleting duplicate: $name ($id)" PREFECT_CLI_PROMPT=false prefect automation delete --id "$id" done log "RECREATE: $name" prefect automation create --from-file "$item_file" fi i=$((i + 1)) done log "Done."
Copy code
---

*2) `list_automations.py`*

```python
import asyncio
import json
import sys

from prefect.client.orchestration import get_client


async def main() -> None:
    async with get_client() as client:
        automations = await client.read_automations()

    result = []
    for automation in automations:
        result.append(
            {
                "id": str(automation.id),
                "name": automation.name,
            }
        )

    json.dump(result, sys.stdout)


if __name__ == "__main__":
    asyncio.run(main())
--- Why this is better * Per-automation temp files are YAML * 
prefect automation create --from-file
and 
prefect automation update --from-file
both work naturally with YAML * Python is isolated to a single, debuggable helper * Shell stays simple and Bullseye-safe * No brittle CLI-table parsing * Input duplicate names are detected up front --- How it works with different input formats This script supports all of these: Single automation:
Copy code
yaml
name: my-automation
trigger:
  type: event
  expect:
    - prefect.flow-run.Failed
actions:
  - type: send-notification
    block_document_id: 12345678-1234-1234-1234-123456789012
    body: hello
List of automations:
Copy code
yaml
- name: a1
  trigger:
    type: event
    expect: [prefect.flow-run.Failed]
  actions: []
- name: a2
  trigger:
    type: event
    expect: [prefect.flow-run.Completed]
  actions: []
Top-level wrapper:
Copy code
yaml
automations:
  - name: a1
    trigger:
      type: event
      expect: [prefect.flow-run.Failed]
    actions: []
  - name: a2
    trigger:
      type: event
      expect: [prefect.flow-run.Completed]
    actions: []
--- How to run it
Copy code
sh
chmod +x prefect-automation-upsert.sh
./prefect-automation-upsert.sh automations.yaml
--- Dependency notes Since you said 
uv
is available, the helper can use your environment via:
Copy code
sh
uv run list_automations.py
If needed, you can make that explicit:
Copy code
sh
uv run --with prefect "$LIST_SCRIPT"
So if your CI image does not already have Prefect available in the uv environment, change this line in the shell script:
Copy code
sh
uv run "$LIST_SCRIPT" > "$EXISTING_JSON"
to:
Copy code
sh
uv run --with prefect "$LIST_SCRIPT" > "$EXISTING_JSON"
--- One optional improvement Right now it updates/creates only the automations present in the file. It does not delete automations in Prefect that are no longer in the file. If you want, I can give you a final version with a 
--prune
flag that also removes previously-managed automations that no longer appear in 
automations.yaml
.
r
@Marvin Yes, please make a final version with --prune. Also run the standard best practice pre-commit checks on the scripts, e.g. use ruff, shellcheck, and so on. And as there are at least two separate files, e.g. the python file and the shell script, and they are getting quite long for Slack's browser chat, can you also put them in an easier to copy or download format?
m
✅ thought for 416.5 seconds
Absolutely — and one important note up front: I can’t actually run 
ruff
, 
shellcheck
, 
shfmt
, or package files for download from here. I can give you: 1. a final multi-file version with 
--prune
2. a copy-pasteable generator script that writes all files locally 3. the exact lint/check commands to run 4. a recommended layout so it’s much easier to copy into your repo than scrolling through giant blobs in Slack Also, I want to correct one detail from the tool output: I do not have a real downloadable tarball to hand you from this chat. So I’ll give you the next best thing: a single setup script that creates the project files exactly on your machine. Recommended project layout
Copy code
text
prefect-automation-upsert/
├── create-project.sh
├── automation_upsert.sh
├── list_automations.py
├── README.md
└── .pre-commit-checks.sh
Below I’m giving you the contents for all four useful files, plus a 
create-project.sh
bootstrapper so you can paste one file and generate the rest. --- 1) 
create-project.sh
 Paste this once, run it, and it will write the other files into 
prefect-automation-upsert/
. ```sh #!/bin/sh set -eu DIR="prefect-automation-upsert" mkdir -p "$DIR" cat > "$DIR/automation_upsert.sh" <<'EOF' #!/bin/sh set -eu # Prefect automation upsert/prune helper # # Usage: # ./automation_upsert.sh automations.yaml [--prune] [--dry-run] [--verbose] # # Behavior: # - 0 existing matches by name -> create # - 1 existing match by name -> update # - >1 existing matches -> delete all duplicates, then create once # - with --prune -> delete automations in Prefect not present in file # # Requirements: # - /bin/sh # - prefect CLI # - jq # - yq # - uv # # Notes: # - Uses separate Python helper 
list_automations.py
# - Designed for Debian Bullseye / POSIX sh compatibility # - Prune matches by name; if you have unrelated automations with the same names, # prune may delete them too. Use carefully. PRUNE=0 DRY_RUN=0 VERBOSE=0 FILE="" log() { printf '%s\n' "$*" >&2 } debug() { if [ "$VERBOSE" -eq 1 ]; then printf '[debug] %s\n' "$*" >&2 fi } die() { log "ERROR: $*" exit 1 } need_cmd() { command -v "$1" >/dev/null 2>&1 || die "Required command not found: $1" } usage() { cat >&2 <<'USAGE' Usage: ./automation_upsert.sh automations.yaml [--prune] [--dry-run] [--verbose] Options: --prune Delete Prefect automations whose names are not in the input file --dry-run Print planned actions without making changes --verbose Enable debug logging USAGE exit 2 } run_cmd() { if [ "$DRY_RUN" -eq 1 ]; then printf '[dry-run] ' >&2 printf '%s ' "$@" >&2 printf '\n' >&2 else "$@" fi } run_delete() { id=$1 if [ "$DRY_RUN" -eq 1 ]; then log "[dry-run] prefect automation delete --id $id" else PREFECT_CLI_PROMPT=false prefect automation delete --id "$id" fi } while [ "$#" -gt 0 ] do case "$1" in --prune) PRUNE=1 ;; --dry-run) DRY_RUN=1 ;; --verbose) VERBOSE=1 ;; -h|--help) usage ;; -*) die "Unknown option: $1" ;; *) if [ -n "$FILE" ]; then die "Only one input file may be provided" fi FILE=$1 ;; esac shift done [ -n "$FILE" ] || usage [ -f "$FILE" ] || die "File not found: $FILE" SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) LIST_SCRIPT="$SCRIPT_DIR/list_automations.py" [ -f "$LIST_SCRIPT" ] || die "Python helper not found: $LIST_SCRIPT" need_cmd prefect need_cmd jq need_cmd yq need_cmd uv need_cmd mktemp need_cmd sort need_cmd uniq TMP_DIR=$(mktemp -d) cleanup() { rm -rf "$TMP_DIR" } trap cleanup EXIT HUP INT TERM ITEMS_DIR="$TMP_DIR/items" mkdir -p "$ITEMS_DIR"
EXISTING_JSON="$TMP_DIR/existing.json" INPUT_NAMES="$TMP_DIR/input-names.txt" INPUT_NAMES_SORTED="$TMP_DIR/input-names-sorted.txt" DUP_NAMES="$TMP_DIR/duplicate-names.txt" debug "Temporary directory: $TMP_DIR" count=$(yq eval ' if type == "!!map" and has("automations") then .automations | length elif type == "!!seq" then length elif type == "!!map" and has("name") then 1 else 0 end ' "$FILE") [ "$count" -gt 0 ] || die "No automations found in $FILE" log "Splitting $FILE into per-automation YAML files ..." i=0 while [ "$i" -lt "$count" ] do out="$ITEMS_DIR/automation-$i.yaml" yq eval " if type == \"!!map\" and has(\"automations\") then .automations[$i] elif type == \"!!seq\" then .[$i] else . end " "$FILE" > "$out" name=$(yq eval -r '.name' "$out") [ "$name" != "null" ] || die "Automation at index $i has no name" printf '%s\n' "$name" >> "$INPUT_NAMES" debug "Wrote $out for automation name: $name" i=$((i + 1)) done sort "$INPUT_NAMES" > "$INPUT_NAMES_SORTED" uniq -d "$INPUT_NAMES_SORTED" > "$DUP_NAMES" || true if [ -s "$DUP_NAMES" ]; then log "Duplicate automation names found in input:" sed 's/^/ - /' "$DUP_NAMES" >&2 die "Input file must not contain duplicate automation names" fi log "Fetching existing automations from Prefect API ..." uv run "$LIST_SCRIPT" > "$EXISTING_JSON" [ -s "$EXISTING_JSON" ] || die "Failed to fetch existing automations" jq -e 'type == "array"' "$EXISTING_JSON" >/dev/null 2>&1 || \ die "Python helper did not return a JSON array" log "Reconciling automations from input ..." i=0 while [ "$i" -lt "$count" ] do item_file="$ITEMS_DIR/automation-$i.yaml" name=$(yq eval -r '.name' "$item_file") matches_file="$TMP_DIR/matches-$i.json" jq --arg name "$name" '[ .[] | select(.name == $name) ]' "$EXISTING_JSON" > "$matches_file" match_count=$(jq 'length' "$matches_file") if [ "$match_count" -eq 0 ]; then log "CREATE: $name" run_cmd prefect automation create --from-file "$item_file" elif [ "$match_count" -eq 1 ]; then id=$(jq -r '.[0].id' "$matches_file") [ "$id" != "null" ] || die "Could not determine id for $name" log "UPDATE: $name ($id)" run_cmd prefect automation update --id "$id" --from-file "$item_file" else log "DEDUPE: $name has $match_count matches; deleting all and recreating once" jq -r '.[].id' "$matches_file" | while IFS= read -r id do [ -n "$id" ] || continue log "Deleting duplicate: $name ($id)" run_delete "$id" done log "RECREATE: $name" run_cmd prefect automation create --from-file "$item_file" fi i=$((i + 1)) done if [ "$PRUNE" -eq 1 ]; then log "Prune enabled: deleting automations not present in input file by name ..." jq -r '.[].name' "$EXISTING_JSON" | sort -u > "$TMP_DIR/existing-names.txt" comm -23 "$TMP_DIR/existing-names.txt" "$INPUT_NAMES_SORTED" > "$TMP_DIR/prune-names.txt" || true if [ -s "$TMP_DIR/prune-names.txt" ]; then while IFS= read -r prune_name do [ -n "$prune_name" ] || continue log "PRUNE NAME: $prune_name" jq --arg name "$prune_name" '.[] | select(.name == $name) | .id' "$EXISTING_JSON" | while IFS= read -r id do [ -n "$id" ] || continue [ "$id" != "null" ] || continue log "Pruning automation: $prune_name ($id)" run_delete "$id" done done < "$TMP_DIR/prune-names.txt" else log "Prune enabled, but no extra automations were found." fi fi log "Done." EOF cat > "$DIR/list_automations.py" <<'EOF' import asyncio import json import sys from prefect.client.orchestration import get_client async def main() -> None: async with get_client() as client:
automations = await client.read_automations() result = [] for automation in automations: result.append( { "id": str(automation.id), "name": automation.name, } ) json.dump(result, sys.stdout) sys.stdout.write("\n") if name == "__main__": asyncio.run(main()) EOF cat > "$DIR/.pre-commit-checks.sh" <<'EOF' #!/bin/sh set -eu DIR=${1:-.} need_cmd() { command -v "$1" >/dev/null 2>&1 || { printf 'Missing required command: %s\n' "$1" >&2 exit 1 } } need_cmd shellcheck need_cmd shfmt need_cmd python3 need_cmd uv printf 'Running shellcheck...\n' shellcheck "$DIR/automation_upsert.sh" "$DIR/create-project.sh" "$DIR/.pre-commit-checks.sh" printf 'Running shfmt...\n' shfmt -d "$DIR/automation_upsert.sh" "$DIR/create-project.sh" "$DIR/.pre-commit-checks.sh" printf 'Running Python compile check...\n' python3 -m py_compile "$DIR/list_automations.py" printf 'Running ruff...\n' uv run --with ruff ruff check "$DIR/list_automations.py" printf 'Done.\n' EOF cat > "$DIR/README.md" <<'EOF' # Prefect automation upsert/prune helper ## Files - 
automation_upsert.sh
— POSIX 
sh
orchestration script - 
list_automations.py
— Python helper to list existing automations from the Prefect API - 
.pre-commit-checks.sh
— local lint/check helper ## Requirements - 
prefect
- 
jq
- 
yq
- 
uv
- 
python3
## Usage
Copy code
sh
./automation_upsert.sh automations.yaml
./automation_upsert.sh automations.yaml --dry-run
./automation_upsert.sh automations.yaml --prune
./automation_upsert.sh automations.yaml --prune --dry-run
## Behavior For each automation in the file: - no existing automation with the same name -> create - exactly one existing automation with the same name -> update - more than one existing automation with the same name -> delete all duplicates and create once If 
--prune
is passed: - any existing Prefect automation whose name does not appear in the input file is deleted ## Important prune caveat Prune is name-based. That means if your workspace contains automations managed outside this file, and they have names not present in this file, they will be deleted by 
--prune
. Use 
--dry-run
first. ## Checks
Copy code
sh
chmod +x ./*.sh
./.pre-commit-checks.sh .
EOF chmod +x "$DIR/automation_upsert.sh" "$DIR/.pre-commit-checks.sh" printf 'Created project in %s\n' "$DIR" printf 'Next steps:\n' printf ' cd %s\n' "$DIR" printf ' chmod +x create-project.sh automation_upsert.sh .pre-commit-checks.sh\n' printf ' ./.pre-commit-checks.sh .\n' EOF ``` --- 2) 
automation_upsert.sh
 If you don’t want the generator and prefer to copy files directly, this is the shell script alone: ```sh #!/bin/sh set -eu PRUNE=0 DRY_RUN=0 VERBOSE=0 FILE="" log() { printf '%s\n' "$*" >&2 } debug() { if [ "$VERBOSE" -eq 1 ]; then printf '[debug] %s\n' "$*" >&2 fi } die() { log "ERROR: $*" exit 1 } need_cmd() { command -v "$1" >/dev/null 2>&1 || die "Required command not found: $1" } usage() { cat >&2 <<'USAGE' Usage: ./automation_upsert.sh automations.yaml [--prune] [--dry-run] [--verbose] Options: --prune Delete Prefect automations whose names are not in the input file --dry-run Print planned actions without making changes --verbose Enable debug logging USAGE exit 2 } run_cmd() { if [ "$DRY_RUN" -eq 1 ]; then printf '[dry-run] ' >&2 printf '%s ' "$@" >&2 printf '\n' >&2 else "$@" fi } run_delete() { id=$1 if [ "$DRY_RUN" -eq 1 ]; then log "[dry-run] prefect automation delete --id $id" else PREFECT_CLI_PROMPT=false prefect automation delete --id "$id" fi } while [ "$#" -gt 0 ] do case "$1" in --prune) PRUNE=1 ;; --dry-run) DRY_RUN=1 ;; --verbose) VERBOSE=1 ;; -h|--help)
usage ;; -*) die "Unknown option: $1" ;; *) if [ -n "$FILE" ]; then die "Only one input file may be provided" fi FILE=$1 ;; esac shift done [ -n "$FILE" ] || usage [ -f "$FILE" ] || die "File not found: $FILE" SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) LIST_SCRIPT="$SCRIPT_DIR/list_automations.py" [ -f "$LIST_SCRIPT" ] || die "Python helper not found: $LIST_SCRIPT" need_cmd prefect need_cmd jq need_cmd yq need_cmd uv need_cmd mktemp need_cmd sort need_cmd uniq TMP_DIR=$(mktemp -d) cleanup() { rm -rf "$TMP_DIR" } trap cleanup EXIT HUP INT TERM ITEMS_DIR="$TMP_DIR/items" mkdir -p "$ITEMS_DIR" EXISTING_JSON="$TMP_DIR/existing.json" INPUT_NAMES="$TMP_DIR/input-names.txt" INPUT_NAMES_SORTED="$TMP_DIR/input-names-sorted.txt" DUP_NAMES="$TMP_DIR/duplicate-names.txt" debug "Temporary directory: $TMP_DIR" count=$(yq eval ' if type == "!!map" and has("automations") then .automations | length elif type == "!!seq" then length elif type == "!!map" and has("name") then 1 else 0 end ' "$FILE") [ "$count" -gt 0 ] || die "No automations found in $FILE" log "Splitting $FILE into per-automation YAML files ..." i=0 while [ "$i" -lt "$count" ] do out="$ITEMS_DIR/automation-$i.yaml" yq eval " if type == \"!!map\" and has(\"automations\") then .automations[$i] elif type == \"!!seq\" then .[$i] else . end " "$FILE" > "$out" name=$(yq eval -r '.name' "$out") [ "$name" != "null" ] || die "Automation at index $i has no name" printf '%s\n' "$name" >> "$INPUT_NAMES" debug "Wrote $out for automation name: $name" i=$((i + 1)) done sort "$INPUT_NAMES" > "$INPUT_NAMES_SORTED" uniq -d "$INPUT_NAMES_SORTED" > "$DUP_NAMES" || true if [ -s "$DUP_NAMES" ]; then log "Duplicate automation names found in input:" sed 's/^/ - /' "$DUP_NAMES" >&2 die "Input file must not contain duplicate automation names" fi log "Fetching existing automations from Prefect API ..." uv run "$LIST_SCRIPT" > "$EXISTING_JSON" [ -s "$EXISTING_JSON" ] || die "Failed to fetch existing automations" jq -e 'type == "array"' "$EXISTING_JSON" >/dev/null 2>&1 || \ die "Python helper did not return a JSON array" log "Reconciling automations from input ..." i=0 while [ "$i" -lt "$count" ] do item_file="$ITEMS_DIR/automation-$i.yaml" name=$(yq eval -r '.name' "$item_file") matches_file="$TMP_DIR/matches-$i.json" jq --arg name "$name" '[ .[] | select(.name == $name) ]' "$EXISTING_JSON" > "$matches_file" match_count=$(jq 'length' "$matches_file") if [ "$match_count" -eq 0 ]; then log "CREATE: $name" run_cmd prefect automation create --from-file "$item_file" elif [ "$match_count" -eq 1 ]; then id=$(jq -r '.[0].id' "$matches_file") [ "$id" != "null" ] || die "Could not determine id for $name" log "UPDATE: $name ($id)" run_cmd prefect automation update --id "$id" --from-file "$item_file" else log "DEDUPE: $name has $match_count matches; deleting all and recreating once" jq -r '.[].id' "$matches_file" | while IFS= read -r id do [ -n "$id" ] || continue log "Deleting duplicate: $name ($id)" run_delete "$id" done log "RECREATE: $name" run_cmd prefect automation create --from-file "$item_file" fi i=$((i + 1)) done if [ "$PRUNE" -eq 1 ]; then log "Prune enabled: deleting automations not present in input file by name ..." jq -r '.[].name' "$EXISTING_JSON" | sort -u > "$TMP_DIR/existing-names.txt" comm -23 "$TMP_DIR/existing-names.txt" "$INPUT_NAMES_SORTED" > "$TMP_DIR/prune-names.txt" || true if [ -s "$TMP_DIR/prune-names.txt" ]; then while IFS= read -r prune_name do
[ -n "$prune_name" ] || continue log "PRUNE NAME: $prune_name" jq --arg name "$prune_name" '.[] | select(.name == $name) | .id' "$EXISTING_JSON" | while IFS= read -r id do [ -n "$id" ] || continue [ "$id" != "null" ] || continue log "Pruning automation: $prune_name ($id)" run_delete "$id" done done < "$TMP_DIR/prune-names.txt" else log "Prune enabled, but no extra automations were found." fi fi log "Done."
Copy code
---

*3) `list_automations.py`*

```python
import asyncio
import json
import sys

from prefect.client.orchestration import get_client


async def main() -> None:
    async with get_client() as client:
        automations = await client.read_automations()

    result = []
    for automation in automations:
        result.append(
            {
                "id": str(automation.id),
                "name": automation.name,
            }
        )

    json.dump(result, sys.stdout)
    sys.stdout.write("\n")


if __name__ == "__main__":
    asyncio.run(main())
--- 4) 
.pre-commit-checks.sh
Copy code
sh
#!/bin/sh
set -eu

DIR=${1:-.}

need_cmd() {
    command -v "$1" >/dev/null 2>&1 || {
        printf 'Missing required command: %s\n' "$1" >&2
        exit 1
    }
}

need_cmd shellcheck
need_cmd shfmt
need_cmd python3
need_cmd uv

printf 'Running shellcheck...\n'
shellcheck "$DIR/automation_upsert.sh" "$DIR/create-project.sh" "$DIR/.pre-commit-checks.sh"

printf 'Running shfmt...\n'
shfmt -d "$DIR/automation_upsert.sh" "$DIR/create-project.sh" "$DIR/.pre-commit-checks.sh"

printf 'Running Python compile check...\n'
python3 -m py_compile "$DIR/list_automations.py"

printf 'Running ruff...\n'
uv run --with ruff ruff check "$DIR/list_automations.py"

printf 'Done.\n'
--- How to use the easier copy/download format I recommend this flow: 1. Paste 
create-project.sh
2. Run:
Copy code
sh
chmod +x create-project.sh
./create-project.sh
cd prefect-automation-upsert
chmod +x ./*.sh
3. Run checks:
Copy code
sh
./.pre-commit-checks.sh .
4. Dry run first:
Copy code
sh
./automation_upsert.sh automations.yaml --dry-run --verbose
5. Then real run:
Copy code
sh
./automation_upsert.sh automations.yaml
6. With prune:
Copy code
sh
./automation_upsert.sh automations.yaml --prune --dry-run
./automation_upsert.sh automations.yaml --prune
--- Important 
--prune
warning This 
--prune
implementation is name-based. That means if your workspace contains: * automations managed by another repo/team * automations not present in this input file then 
--prune
will delete them if their names are absent from the file. So the safe rollout is:
Copy code
sh
./automation_upsert.sh automations.yaml --prune --dry-run --verbose
first, inspect the planned deletes, then run it for real. --- Best-practice checks I recommend Install tools if needed:
Copy code
sh
apt-get update
apt-get install -y shellcheck
uv tool install shfmt
Then run:
Copy code
sh
./.pre-commit-checks.sh .
If you want, I can do one more pass and make this safer still by adding: * a required 
--apply
flag for destructive actions * a backup export before prune/delete * a managed-by tag filter so prune only touches automations marked as managed by this repo That would be the version I’d trust most in production.
r
@Marvin you've forgotten that it should work in POSIX
m
✅ thought for 36.9 seconds
You're right — thanks for catching that. The mistake is that the latest script uses 
comm
, which is not guaranteed by POSIX. Also, relying on assorted GNU-ish conveniences without checking was sloppy if the goal is truly POSIX-oriented/Bullseye-safe. Let me fix that properly. What I’m changing * keep 
/bin/sh
* keep separate Python helper * keep YAML splitting with 
yq
* keep 
--prune
* remove the non-POSIX 
comm
dependency * avoid other unnecessary non-POSIX assumptions in the shell logic The cleanest POSIX-safe way to do prune is: * build a sorted/unique input-name list * iterate existing automations from JSON * for each existing name, use 
grep -F -x
against the input-name file * if absent, prune its IDs That avoids 
comm
. Below is a corrected final version. --- 
automation_upsert.sh
 ```sh #!/bin/sh set -eu PRUNE=0 DRY_RUN=0 VERBOSE=0 FILE="" log() { printf '%s\n' "$*" >&2 } debug() { if [ "$VERBOSE" -eq 1 ]; then printf '[debug] %s\n' "$*" >&2 fi } die() { log "ERROR: $*" exit 1 } need_cmd() { command -v "$1" >/dev/null 2>&1 || die "Required command not found: $1" } usage() { cat >&2 <<'USAGE' Usage: ./automation_upsert.sh automations.yaml [--prune] [--dry-run] [--verbose] Options: --prune Delete Prefect automations whose names are not in the input file --dry-run Print planned actions without making changes --verbose Enable debug logging USAGE exit 2 } run_cmd() { if [ "$DRY_RUN" -eq 1 ]; then printf '[dry-run]' >&2 for arg in "$@" do printf ' %s' "$arg" >&2 done printf '\n' >&2 else "$@" fi } run_delete() { id=$1 if [ "$DRY_RUN" -eq 1 ]; then log "[dry-run] prefect automation delete --id $id" else PREFECT_CLI_PROMPT=false prefect automation delete --id "$id" fi } while [ "$#" -gt 0 ] do case "$1" in --prune) PRUNE=1 ;; --dry-run) DRY_RUN=1 ;; --verbose) VERBOSE=1 ;; -h|--help) usage ;; -*) die "Unknown option: $1" ;; *) if [ -n "$FILE" ]; then die "Only one input file may be provided" fi FILE=$1 ;; esac shift done [ -n "$FILE" ] || usage [ -f "$FILE" ] || die "File not found: $FILE" SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) LIST_SCRIPT="$SCRIPT_DIR/list_automations.py" [ -f "$LIST_SCRIPT" ] || die "Python helper not found: $LIST_SCRIPT" need_cmd prefect need_cmd jq need_cmd yq need_cmd uv need_cmd mktemp need_cmd sort need_cmd uniq need_cmd grep TMP_DIR=$(mktemp -d) cleanup() { rm -rf "$TMP_DIR" } trap cleanup EXIT HUP INT TERM ITEMS_DIR="$TMP_DIR/items" mkdir -p "$ITEMS_DIR" EXISTING_JSON="$TMP_DIR/existing.json" INPUT_NAMES="$TMP_DIR/input-names.txt" INPUT_NAMES_SORTED="$TMP_DIR/input-names-sorted.txt" DUP_NAMES="$TMP_DIR/duplicate-names.txt" EXISTING_NAMES="$TMP_DIR/existing-names.txt" debug "Temporary directory: $TMP_DIR" count=$(yq eval ' if type == "!!map" and has("automations") then .automations | length elif type == "!!seq" then length elif type == "!!map" and has("name") then 1 else 0 end ' "$FILE") [ "$count" -gt 0 ] || die "No automations found in $FILE" log "Splitting $FILE into per-automation YAML files ..." i=0 while [ "$i" -lt "$count" ] do out="$ITEMS_DIR/automation-$i.yaml" yq eval " if type == \"!!map\" and has(\"automations\") then .automations[$i] elif type == \"!!seq\" then .[$i] else . end " "$FILE" > "$out" name=$(yq eval -r '.name' "$out") [ "$name" != "null" ] || die "Automation at index $i has no name" printf '%s\n' "$name" >> "$INPUT_NAMES" debug "Wrote $out for automation name: $name" i=$((i + 1)) done
sort "$INPUT_NAMES" > "$INPUT_NAMES_SORTED" uniq -d "$INPUT_NAMES_SORTED" > "$DUP_NAMES" || true if [ -s "$DUP_NAMES" ]; then log "Duplicate automation names found in input:" sed 's/^/ - /' "$DUP_NAMES" >&2 die "Input file must not contain duplicate automation names" fi log "Fetching existing automations from Prefect API ..." uv run "$LIST_SCRIPT" > "$EXISTING_JSON" [ -s "$EXISTING_JSON" ] || die "Failed to fetch existing automations" jq -e 'type == "array"' "$EXISTING_JSON" >/dev/null 2>&1 || \ die "Python helper did not return a JSON array" log "Reconciling automations from input ..." i=0 while [ "$i" -lt "$count" ] do item_file="$ITEMS_DIR/automation-$i.yaml" name=$(yq eval -r '.name' "$item_file") matches_file="$TMP_DIR/matches-$i.json" jq --arg name "$name" '[ .[] | select(.name == $name) ]' "$EXISTING_JSON" > "$matches_file" match_count=$(jq 'length' "$matches_file") if [ "$match_count" -eq 0 ]; then log "CREATE: $name" run_cmd prefect automation create --from-file "$item_file" elif [ "$match_count" -eq 1 ]; then id=$(jq -r '.[0].id' "$matches_file") [ "$id" != "null" ] || die "Could not determine id for $name" log "UPDATE: $name ($id)" run_cmd prefect automation update --id "$id" --from-file "$item_file" else log "DEDUPE: $name has $match_count matches; deleting all and recreating once" jq -r '.[].id' "$matches_file" | while IFS= read -r id do [ -n "$id" ] || continue [ "$id" != "null" ] || continue log "Deleting duplicate: $name ($id)" run_delete "$id" done log "RECREATE: $name" run_cmd prefect automation create --from-file "$item_file" fi i=$((i + 1)) done if [ "$PRUNE" -eq 1 ]; then log "Prune enabled: deleting automations not present in input file by name ..." jq -r '.[].name' "$EXISTING_JSON" | sort | uniq > "$EXISTING_NAMES" while IFS= read -r existing_name do [ -n "$existing_name" ] || continue if grep -F -x -q -- "$existing_name" "$INPUT_NAMES_SORTED"; then debug "Keeping existing automation name present in input: $existing_name" else log "PRUNE NAME: $existing_name" jq --arg name "$existing_name" '.[] | select(.name == $name) | .id' "$EXISTING_JSON" | while IFS= read -r id do [ -n "$id" ] || continue [ "$id" != "null" ] || continue log "Pruning automation: $existing_name ($id)" run_delete "$id" done fi done < "$EXISTING_NAMES" fi log "Done."
Copy code
---

*`list_automations.py`*

```python
import asyncio
import json
import sys

from prefect.client.orchestration import get_client


async def main() -> None:
    async with get_client() as client:
        automations = await client.read_automations()

    result = []
    for automation in automations:
        result.append(
            {
                "id": str(automation.id),
                "name": automation.name,
            }
        )

    json.dump(result, sys.stdout)
    sys.stdout.write("\n")


if __name__ == "__main__":
    asyncio.run(main())
--- POSIX notes This shell version now avoids: * 
[[ ... ]]
* arrays * process substitution * 
pipefail
* 
comm
It uses only normal POSIX shell constructs plus external commands like: * 
grep
* 
sort
* 
uniq
* 
mktemp
* 
jq
* 
yq
* 
uv
Strictly speaking, 
mktemp
is not in the POSIX base spec either, but on Debian Bullseye it is standard and appropriate. If you want fullest possible portability beyond Bullseye, I can replace 
mktemp
with a more old-school temp-dir pattern. --- Updated checks For the shell script, the best practice checks are still: ```sh shellcheck automation_upsert.sh shfmt -w automation_upsert.sh python3 -m py_compile list_automations.py
uv run --with ruff ruff check list_automations.py ``` If you want, I can do one final pass and give you: * a corrected POSIX-safe 
create-project.sh
* a tiny 
Makefile
* a 
pre-commit-config.yaml
so the whole thing is easier to drop straight into your repo.
2 Views
Open in Slack
PreviousNext
Powered by