OTA phase 5: tools and docs ready for the public export
pub-release / release (push) Successful in 32m57s
pub-release / release (push) Successful in 32m57s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
97618f223f
commit
d22005a6fd
12 files changed
+704
-24
No files matched your search
@@ -204,7 +204,7 @@ zůstane v bootloaderu a vrátí se přes USB/BLE DFU.
|
||||
Nástroje: `tools/ota_push.py` (USB), `tools/ota_mesh_push.py` (přes mesh
|
||||
skrz companion; potřebuje pythonovou knihovnu `meshcore`, admin heslo bere
|
||||
z `[secrets] admin_pw` v `platformio.local.ini`), `tools/ota_pack.py info`.
|
||||
Postup a čísla: [docs/ota-admin.md](docs/ota-admin.md).
|
||||
Postup a čísla: [docs/ota.md](docs/ota.md).
|
||||
* **Experimentální**: vyzkoušeno na XIAO na lavici přes USB i přes mesh
|
||||
(1 hop). Vadný nový firmware nemá kam se vrátit (není druhá banka) – nový
|
||||
release nejdřív na jednom uzlu na dosah. Stojí ~10 kB RAM navíc.
|
||||
@@ -424,7 +424,7 @@ cut during the rewrite leaves it in the bootloader, back via USB/BLE DFU.
|
||||
mesh through a companion; needs the `meshcore` Python library, takes the
|
||||
admin password from `[secrets] admin_pw` in `platformio.local.ini`),
|
||||
`tools/ota_pack.py info`. How it works and bench numbers:
|
||||
[docs/ota-admin.md](docs/ota-admin.md).
|
||||
[docs/ota.md](docs/ota.md).
|
||||
* **Experimental**: tried on XIAO boards on the bench over USB and over the
|
||||
mesh (1 hop). A bad new firmware has nothing to fall back to (no second
|
||||
bank) -- try a new release on one node within reach first. Costs ~10 kB of
|
||||
|
||||
+108
@@ -0,0 +1,108 @@
|
||||
# Admin-pushed firmware update (experimental `-ota` images)
|
||||
|
||||
The `*-ota` and `*-observer-ota` images of this fork's releases can take a new
|
||||
firmware from their admin -- over the USB or BLE console, or **over the mesh,
|
||||
as remote admin CLI**. Nothing changes in the bootloader and there is no second
|
||||
bank: the node stores the package in free flash, checks all of it, and only on
|
||||
an explicit `fw apply` rewrites itself from a routine running in RAM.
|
||||
|
||||
The point is the **delta**: a package against the image the node runs is
|
||||
usually a few kB, so it goes through LoRa in minutes. A full image (~300 kB)
|
||||
goes only over USB or BLE.
|
||||
|
||||
**Experimental.** Tried on XIAO nRF52840 boards over USB and over the mesh
|
||||
(one hop). A broken new firmware has nothing to fall back to: try every
|
||||
release on one node within reach before the ones you cannot reach.
|
||||
|
||||
## What a release carries
|
||||
|
||||
For every `<board>_<role>-ota` / `-observer-ota` image:
|
||||
|
||||
| file | what |
|
||||
|---|---|
|
||||
| `<image>-<release>.zip` / `.uf2` | the image, flashed the usual way (first time) |
|
||||
| `<image>-<release>.mcfw` | full package: USB/BLE console only |
|
||||
| `<image>-<release>.from-<earlier>.mcfw` | delta from an earlier release's same image |
|
||||
| `manifest.json` | image sha256, target, build epoch, packages, deltas by base |
|
||||
| `SHA256SUMS`, `SHA256SUMS.sig` | checksums, signed by `meshcore-ota@tth-pub` |
|
||||
|
||||
Check a download:
|
||||
|
||||
ssh-keygen -Y verify -f tools/ota_allowed_signers -I meshcore-ota@tth-pub \
|
||||
-n meshcore-ota -s SHA256SUMS.sig < SHA256SUMS
|
||||
sha256sum -c --ignore-missing SHA256SUMS
|
||||
|
||||
`tools/ota_pack.py fetch <image> --base <sha8>` does both and picks the delta
|
||||
for the node's image (`--full` for the full package, `--release <tag>` for
|
||||
another than the latest).
|
||||
|
||||
## On the node
|
||||
|
||||
```
|
||||
fw status state, battery, the running image's size and sha256 prefix
|
||||
fw ver target and build epoch a package for this image needs
|
||||
fw begin <bytes> [sid] start a transfer
|
||||
fw w <offset> <base64> one 96 B chunk (text)
|
||||
fw missing what is still missing
|
||||
fw verify queue the full check, poll fw status
|
||||
fw apply battery gate, then rewrite and reboot
|
||||
fw apply downgrade local console only: an older (or the same) image
|
||||
fw abort
|
||||
```
|
||||
|
||||
What the node checks before it touches its own image:
|
||||
|
||||
1. The **manifest**, as soon as it is in: an Ed25519 signature by the key the
|
||||
image was built with (`[ota] pubkey`), the target (board + variant: an
|
||||
`-observer-ota` node takes no `-ota` package), the size, for a delta the
|
||||
base (the sha256 of the image it runs), and the build epoch.
|
||||
2. Over the mesh: **only a delta, and only a newer build.** An older or equal
|
||||
one is refused right after the manifest; a full image is refused when the
|
||||
transfer begins. Locally a full image is fine, and an older one takes
|
||||
`fw apply downgrade`.
|
||||
3. `fw verify`: the payload hash, and a delta applied **dry** -- the same code
|
||||
the real run uses, with the flash untouched -- must produce the signed
|
||||
image hash.
|
||||
4. `fw apply`: battery at 3.8 V or more unless the node runs on USB. The
|
||||
bootloader settings get the new image's CRC **first**, so a power cut during
|
||||
the rewrite leaves the node in the bootloader (back with USB/BLE DFU), not
|
||||
in half an image.
|
||||
|
||||
## Tools
|
||||
|
||||
```bash
|
||||
# a delta for the node's image, from the latest release
|
||||
tools/ota_pack.py fetch Xiao_nrf52_repeater-ota --base <sha8 from fw status>
|
||||
|
||||
# over the mesh, through a companion (USB port or tcp://host:port), as admin
|
||||
tools/ota_mesh_push.py /dev/serial/by-id/<companion> <node pubkey prefix> pkg.mcfw --apply
|
||||
|
||||
# over the node's own USB console
|
||||
tools/ota_push.py /dev/serial/by-id/<node> pkg.mcfw --apply
|
||||
```
|
||||
|
||||
`ota_mesh_push.py` needs the `meshcore` Python library. It logs in as admin
|
||||
(password from `[secrets] admin_pw` of `platformio.local.ini`, or asked for),
|
||||
sends binary chunks with a pause between them, asks which are missing and
|
||||
resends only those, and keeps an airtime budget (`--duty`, default 1 % per
|
||||
hour: the 865-868 MHz band). A refused package costs the two manifest chunks,
|
||||
not the whole transfer. Check `get radio` on both ends first: this transmits.
|
||||
|
||||
## Your own builds
|
||||
|
||||
A build with `-D FW_OTA=1` trusts `[ota] pubkey` of `platformio.ini` -- the
|
||||
release key. To sign your own packages, set your key in the gitignored
|
||||
`platformio.local.ini`:
|
||||
|
||||
```ini
|
||||
[ota]
|
||||
pubkey = <tools/ota_pack.py pubkey --key your.key>
|
||||
```
|
||||
|
||||
and pack with `tools/ota_pack.py pack|diff <env> --key your.key`. Target and
|
||||
build epoch come from the ELF, never from the command line. Keep every image
|
||||
you flash: a delta needs the exact bytes the node runs as its base.
|
||||
|
||||
Cost: about 10 kB of RAM on a repeater, the applier code in RAM, a 32 kB
|
||||
buffer while a full package is verified or applied. Classic images (without
|
||||
`-ota`) have none of it.
|
||||
+1
-1
@@ -25,7 +25,7 @@ extra_configs =
|
||||
ble_pin = 123456
|
||||
admin_pw = password
|
||||
|
||||
; Admin-pushed OTA (src/helpers/FwUpdate, docs/ota-admin.md): the PUBLIC key
|
||||
; Admin-pushed OTA (src/helpers/FwUpdate, docs/ota.md): the PUBLIC key
|
||||
; every OTA package signature is checked against, raw Ed25519 hex. Default: the
|
||||
; public OTA key meshcore-ota@tth-pub, the one the -ota release images are
|
||||
; signed with (tools/ota_allowed_signers has it as an ssh key, for the release
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
#pragma once
|
||||
|
||||
// CUSTOM (TeTeHacko): firmware update pushed by an admin, no bootloader change.
|
||||
// Design and the reasons behind it: docs/ota-admin.md.
|
||||
// Design and the reasons behind it: docs/ota.md.
|
||||
//
|
||||
// Two package formats, one signed 256 B manifest:
|
||||
// 1 deflate-compressed full image, staged in the free flash above the running
|
||||
|
||||
@@ -43,6 +43,19 @@ a different board.
|
||||
| `ble_dfu.py` | legacy Nordic DFU over BLE for the Adafruit nRF52 bootloader (buttonless, both phases) |
|
||||
| `build_version.py` | PlatformIO pre-script: real version and build time for `ver` |
|
||||
|
||||
## Admin-pushed firmware update (experimental `-ota` images)
|
||||
|
||||
| tool | what |
|
||||
|---|---|
|
||||
| `ota_pack.py` | `fetch` a signed package of a release and check it; `info`; build and sign your own (`pack`, `diff`) |
|
||||
| `ota_push.py` | push a package over the USB console (text, or the mesh protocol via `fw bin`) |
|
||||
| `ota_mesh_push.py` | push a delta over the mesh through a companion, as admin |
|
||||
| `ota_ramlib.py` | PlatformIO script: runs the delta applier from RAM (a build dependency) |
|
||||
| `ota_check_ramfunc.sh` | checks that an `-ota` build's RAM code never calls into flash |
|
||||
| `ota_allowed_signers` | the release signer (`ssh-keygen -Y verify`) |
|
||||
|
||||
How it works: [docs/ota.md](../docs/ota.md).
|
||||
|
||||
## Bench and diagnostics
|
||||
|
||||
| tool | what |
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
# SPDX-License-Identifier: MIT
|
||||
meshcore-fw@tth namespaces="meshcore-fw" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAOJAwaP/iddhVlx07ZUlaE2+Ygl0yGTpiaHEe74f/+Q
|
||||
Executable
+68
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env bash
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Checks that the FwUpdate apply paths really run from RAM only: the inflate copy
|
||||
# (RAMFUNC in src/helpers/FwUpdate.cpp) and the delta applier (FwPatch.cpp with
|
||||
# the vendored detools, moved to RAM by tools/ota_ramlib.py). EVERY function
|
||||
# that lives in RAM is checked: each direct branch (bl, b, b.w) must land in
|
||||
# RAM, and no literal may point into the flash the apply erases. GCC silently
|
||||
# moved the never-written inflate tables to flash once -- this is the guard for
|
||||
# that, and for a libc call slipping into the vendored code.
|
||||
#
|
||||
# tools/ota_check_ramfunc.sh <env>
|
||||
set -euo pipefail
|
||||
|
||||
case "${1:-}" in
|
||||
-h | --help | "")
|
||||
sed -n '3,/^set -euo/{/^set -euo/d;s/^# \{0,1\}//;p}' "$0"
|
||||
[[ -n "${1:-}" ]] && exit 0 || exit 2
|
||||
;;
|
||||
esac
|
||||
env="$1"
|
||||
elf=".pio/build/${env}/firmware.elf"
|
||||
bin="${HOME}/.platformio/packages/toolchain-gccarmnoneeabi/bin"
|
||||
[[ -f "$elf" ]] || {
|
||||
echo "missing $elf"
|
||||
exit 2
|
||||
}
|
||||
|
||||
bad=0
|
||||
count=0
|
||||
while read -r addr size _type name; do
|
||||
start=$((16#$addr))
|
||||
end=$((start + 16#$size))
|
||||
count=$((count + 1))
|
||||
dis=$("$bin/arm-none-eabi-objdump" -d -j .data \
|
||||
--start-address="$start" --stop-address="$end" "$elf")
|
||||
while read -r target; do
|
||||
if ((16#$target < 0x20000000)); then
|
||||
echo "FAIL $name calls flash 0x$target"
|
||||
bad=1
|
||||
fi
|
||||
done < <(grep -oP '\s(bl|b|b\.w|bl\.w|b\.n)\s+\K[0-9a-f]+(?=\s+<)' <<<"$dis" || true)
|
||||
while read -r word; do
|
||||
v=$((word))
|
||||
if ((v >= 0x27000 && v < 0x100000 && v != 0xFF000)); then
|
||||
echo "FAIL $name uses flash address $word"
|
||||
bad=1
|
||||
fi
|
||||
# literal words; the halfword offsets of an inline tbb/tbh jump table that
|
||||
# objdump prints as .word are skipped (they are not addresses)
|
||||
done < <(awk '/\ttb[bh]\t/ { skip = 1; next }
|
||||
/\t\.(word|short|byte)\t/ { if (!skip) { sub(/.*\.word\t/, ""); if ($0 ~ /^0x/) print $1 } ; next }
|
||||
{ skip = 0 }' <<<"$dis")
|
||||
done < <("$bin/arm-none-eabi-nm" -S "$elf" | awk '$3 ~ /^[tT]$/ && $1 ~ /^2000/')
|
||||
|
||||
ram_syms=$("$bin/arm-none-eabi-nm" "$elf" | awk '$1 ~ /^2000/ {print $3}')
|
||||
for must in fwp_run cb_read heatshrink_decoder_poll patch_and_reset copy_and_reset inf_run; do
|
||||
if ! grep -qE "^(_ZL[0-9]+)?${must}([A-Za-z0-9_.]*)?$" <<<"$ram_syms"; then
|
||||
echo "FAIL $must is not in RAM"
|
||||
bad=1
|
||||
fi
|
||||
done
|
||||
|
||||
if ((count < 10)); then
|
||||
echo "FAIL only $count RAM functions found, symbol names changed?"
|
||||
exit 1
|
||||
fi
|
||||
if ((bad)); then exit 1; fi
|
||||
echo "ok: $count RAM functions, no call or pointer into flash"
|
||||
Executable
+102
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: MIT
|
||||
"""Regenerate test/test_fwpatch/fixtures.h (in-place patches for the native test).
|
||||
|
||||
tools/ota_fixtures.py # writes test/test_fwpatch/fixtures.h
|
||||
|
||||
Needs detools (pip, in a venv; not on the CI runner -- that is why the result is
|
||||
committed). The images themselves are NOT stored: test_fwpatch.cpp rebuilds them
|
||||
with the same xorshift32 generator as make_images() below, so only the patches
|
||||
(a few kB) live in the header. Change the generator on both sides or neither.
|
||||
|
||||
Cases:
|
||||
grow new = old + 8 kB inserted early, memory tight enough that the
|
||||
shifted old image and the new one overlap (the hard path)
|
||||
shrink new = old - 3 kB, scattered edits
|
||||
same new = old with 16 changed bytes (the "rebuild" case)
|
||||
"""
|
||||
|
||||
import io
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
if any(a in ("-h", "--help") for a in sys.argv[1:]):
|
||||
print(__doc__)
|
||||
sys.exit(0)
|
||||
|
||||
import detools
|
||||
|
||||
OUT = pathlib.Path(__file__).resolve().parent.parent / "test/test_fwpatch/fixtures.h"
|
||||
PAGE = 4096
|
||||
|
||||
|
||||
def xorshift(seed: int, n: int) -> bytes:
|
||||
x = seed & 0xFFFFFFFF
|
||||
out = bytearray(n)
|
||||
for i in range(n):
|
||||
x ^= (x << 13) & 0xFFFFFFFF
|
||||
x ^= x >> 17
|
||||
x ^= (x << 5) & 0xFFFFFFFF
|
||||
out[i] = x & 0xFF
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def make_images(case: str) -> tuple[bytes, bytes]:
|
||||
old = xorshift(0x12345678, 60000)
|
||||
if case == "grow":
|
||||
new = old[:5000] + xorshift(0xCAFEBABE, 8192) + old[5000:]
|
||||
elif case == "shrink":
|
||||
b = bytearray(old[:20000] + old[23000:])
|
||||
for i in range(0, len(b), 4093):
|
||||
b[i] ^= 0x5A
|
||||
new = bytes(b)
|
||||
else: # same
|
||||
b = bytearray(old)
|
||||
for i in range(16):
|
||||
b[1000 + i * 3001] ^= 0xFF
|
||||
new = bytes(b)
|
||||
return old, new
|
||||
|
||||
|
||||
MEMORY = {"grow": 30 * PAGE, "shrink": 32 * PAGE, "same": 40 * PAGE}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
lines = [
|
||||
"// GENERATED by tools/ota_fixtures.py -- do not edit.",
|
||||
"#pragma once",
|
||||
"#include <stdint.h>",
|
||||
"",
|
||||
]
|
||||
for case, mem in MEMORY.items():
|
||||
old, new = make_images(case)
|
||||
p = io.BytesIO()
|
||||
detools.create_patch(
|
||||
io.BytesIO(old),
|
||||
io.BytesIO(new),
|
||||
p,
|
||||
compression="heatshrink",
|
||||
patch_type="in-place",
|
||||
memory_size=mem,
|
||||
segment_size=PAGE,
|
||||
heatshrink_window_sz2=11,
|
||||
heatshrink_lookahead_sz2=10,
|
||||
)
|
||||
data = p.getvalue()
|
||||
lines.append(f"// {case}: old {len(old)} B, new {len(new)} B, memory {mem} B")
|
||||
lines.append(f"static const uint32_t fx_{case}_mem = {mem};")
|
||||
lines.append(f"static const uint32_t fx_{case}_old = {len(old)};")
|
||||
lines.append(f"static const uint32_t fx_{case}_new = {len(new)};")
|
||||
lines.append(f"static const uint8_t fx_{case}_patch[{len(data)}] = {{")
|
||||
for i in range(0, len(data), 16):
|
||||
lines.append(" " + ", ".join(f"0x{b:02x}" for b in data[i : i + 16]) + ",")
|
||||
lines.append("};")
|
||||
lines.append("")
|
||||
print(f"{case}: patch {len(data)} B")
|
||||
OUT.write_text("\n".join(lines))
|
||||
print(f"wrote {OUT}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+10
-3
@@ -1,4 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: MIT
|
||||
"""Push an OTA package (tools/ota_pack.py) to a node OVER THE MESH, as admin.
|
||||
|
||||
tools/ota_mesh_push.py <companion> <dest> <package> [--apply] [--duty 1]
|
||||
@@ -39,8 +40,8 @@ Every `fw w` is idempotent on the node, so a lost reply is simply resent; the
|
||||
node checks the manifest as soon as its chunks arrive and answers ERR for a
|
||||
wrong board or base -- that stops the transfer.
|
||||
|
||||
The admin password comes from platformio.local.ini ([secrets] admin_pw) and is
|
||||
never printed. CHECK `get radio` on both ends first: this transmits.
|
||||
The admin password comes from platformio.local.ini ([secrets] admin_pw), or is
|
||||
asked for on the terminal; it is never printed. CHECK `get radio` on both ends first: this transmits.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
@@ -97,11 +98,17 @@ def w_payload(sid: int, off: int, chunk: bytes) -> bytes:
|
||||
|
||||
|
||||
def admin_pw() -> str:
|
||||
"""[secrets] admin_pw of platformio.local.ini, else asked on the terminal
|
||||
(never an argument: it would show in the process list and the history)."""
|
||||
cp = configparser.ConfigParser(interpolation=None, strict=False)
|
||||
cp.read(INI)
|
||||
pw = cp.get("secrets", "admin_pw", fallback="").strip()
|
||||
if not pw and sys.stdin.isatty():
|
||||
import getpass
|
||||
|
||||
pw = getpass.getpass("admin password of the node: ").strip()
|
||||
if not pw:
|
||||
sys.exit(f"no [secrets] admin_pw in {INI}")
|
||||
sys.exit(f"no admin password: [secrets] admin_pw in {INI}, or run it on a terminal")
|
||||
return pw
|
||||
|
||||
|
||||
|
||||
+69
-17
@@ -1,4 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: MIT
|
||||
"""Pack a firmware image for the admin-pushed OTA (src/helpers/FwUpdate.*).
|
||||
|
||||
ota_pack.py keygen [--key PATH]
|
||||
@@ -14,11 +15,14 @@
|
||||
ota_pack.py archive <env>
|
||||
store the env's image in the archive (pack and diff do it too)
|
||||
ota_pack.py info <file.mcfw> [--pubkey HEX]
|
||||
ota_pack.py fetch <env> --release fw-<sha8> (--base <sha-prefix> | --full) [--out FILE]
|
||||
download a package that CI built (tth-fw-release.yml) and check it: the
|
||||
ssh signature of SHA256SUMS (meshcore-fw@tth, tools/fw_allowed_signers),
|
||||
ota_pack.py fetch <artifact> [--release TAG|latest] (--base <sha-prefix> | --full) [--out FILE]
|
||||
download an OTA package of a public release and check it: the ssh
|
||||
signature of SHA256SUMS (meshcore-ota@tth-pub, tools/ota_allowed_signers),
|
||||
the file's sha256, and the manifest signature against [ota] pubkey.
|
||||
<artifact> is the image's name in the release, e.g.
|
||||
Xiao_nrf52_repeater-ota or SenseCap_Solar_repeater-observer-ota.
|
||||
--base picks the delta for the image the node runs (`fw status`).
|
||||
--private: our mast releases fw-<sha8> instead (token, fleet signer).
|
||||
|
||||
Image archive: ~/.local/share/meshcore-ota/images/<sha256>.bin -- every image
|
||||
that may become a base has to land there BEFORE it is flashed by other means
|
||||
@@ -51,6 +55,7 @@ import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
@@ -76,9 +81,21 @@ MANIFEST = 256
|
||||
PAGE = 4096
|
||||
HS_WINDOW, HS_LOOKAHEAD = 11, 10
|
||||
ARCHIVE = Path.home() / ".local/share/meshcore-ota/images"
|
||||
API = "https://git.rfa.cz/api/v1/repos/tth/meshcore-tth"
|
||||
SIGNERS = REPO / "tools/fw_allowed_signers"
|
||||
SIGNER, NAMESPACE = "meshcore-fw@tth", "meshcore-fw"
|
||||
# where `fetch` looks: the public releases, or (--private) our mast releases
|
||||
SOURCES = {
|
||||
"public": {
|
||||
"api": "https://git.rfa.cz/api/v1/repos/tth-pub/meshcore-tth",
|
||||
"signers": REPO / "tools/ota_allowed_signers",
|
||||
"signer": "meshcore-ota@tth-pub",
|
||||
"namespace": "meshcore-ota",
|
||||
},
|
||||
"private": {
|
||||
"api": "https://git.rfa.cz/api/v1/repos/tth/meshcore-tth",
|
||||
"signers": REPO / "tools/fw_allowed_signers",
|
||||
"signer": "meshcore-fw@tth",
|
||||
"namespace": "meshcore-fw",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def load_key(path: Path) -> Ed25519PrivateKey:
|
||||
@@ -363,15 +380,35 @@ def release_pubkey() -> str | None:
|
||||
|
||||
|
||||
def cmd_fetch(args: argparse.Namespace) -> None:
|
||||
tag = args.release if args.release.startswith("fw-") else f"fw-{args.release[:8]}"
|
||||
tok = gitea_token()
|
||||
src = SOURCES["private" if args.private else "public"]
|
||||
api = str(src["api"])
|
||||
# the public releases are readable without a token, and get none
|
||||
headers = {"Authorization": f"token {gitea_token()}"} if args.private else {}
|
||||
|
||||
def get(url: str) -> bytes:
|
||||
req = urllib.request.Request(url, headers={"Authorization": f"token {tok}"})
|
||||
req = urllib.request.Request(url, headers=headers)
|
||||
with urllib.request.urlopen(req, timeout=60) as r:
|
||||
return r.read()
|
||||
|
||||
rel = json.loads(get(f"{API}/releases/tags/{tag}"))
|
||||
if args.private:
|
||||
tag = (
|
||||
args.release if args.release.startswith("fw-") else f"fw-{args.release[:8]}"
|
||||
)
|
||||
rel = json.loads(get(f"{api}/releases/tags/{tag}"))
|
||||
elif args.release == "latest":
|
||||
rels = [
|
||||
r
|
||||
for r in json.loads(get(f"{api}/releases?limit=20"))
|
||||
if "+tth." in r["tag_name"]
|
||||
and any(a["name"] == "manifest.json" for a in r.get("assets", []))
|
||||
]
|
||||
if not rels:
|
||||
sys.exit("no public release with OTA packages yet")
|
||||
rel = max(rels, key=lambda r: r["created_at"])
|
||||
tag = rel["tag_name"]
|
||||
else:
|
||||
tag = args.release
|
||||
rel = json.loads(get(f"{api}/releases/tags/{urllib.parse.quote(tag, safe='')}"))
|
||||
assets = {a["name"]: a["browser_download_url"] for a in rel.get("assets", [])}
|
||||
for name in ("SHA256SUMS", "SHA256SUMS.sig", "manifest.json"):
|
||||
if name not in assets:
|
||||
@@ -386,11 +423,11 @@ def cmd_fetch(args: argparse.Namespace) -> None:
|
||||
"-Y",
|
||||
"verify",
|
||||
"-f",
|
||||
str(args.allowed_signers),
|
||||
str(args.allowed_signers or src["signers"]),
|
||||
"-I",
|
||||
SIGNER,
|
||||
str(src["signer"]),
|
||||
"-n",
|
||||
NAMESPACE,
|
||||
str(src["namespace"]),
|
||||
"-s",
|
||||
sig.name,
|
||||
],
|
||||
@@ -437,7 +474,8 @@ def cmd_fetch(args: argparse.Namespace) -> None:
|
||||
)
|
||||
sys.exit(
|
||||
f"{tag}: {len(hits)} deltas for base {spec}; built for: {have or 'none'}"
|
||||
" -- rerun tth-fw-release with delta_from, or --full"
|
||||
" -- a release has deltas only from the earlier releases it was built"
|
||||
" against (tth-fw-release: delta_from); otherwise --full over USB/BLE"
|
||||
)
|
||||
name = hits[0]["file"]
|
||||
data = checked(name)
|
||||
@@ -483,8 +521,19 @@ def main() -> None:
|
||||
p.add_argument("file")
|
||||
p.add_argument("--pubkey")
|
||||
p = sub.add_parser("fetch")
|
||||
p.add_argument("env")
|
||||
p.add_argument("--release", required=True, help="fw-<sha8> or a commit sha")
|
||||
p.add_argument(
|
||||
"env", help="artifact, e.g. Xiao_nrf52_repeater-ota (--private: the env)"
|
||||
)
|
||||
p.add_argument(
|
||||
"--release",
|
||||
default="latest",
|
||||
help="public tag (v1.18.0+tth.YYYYMMDD) or latest; --private: fw-<sha8>",
|
||||
)
|
||||
p.add_argument(
|
||||
"--private",
|
||||
action="store_true",
|
||||
help="our mast releases (tth/meshcore-tth, token, fleet signer)",
|
||||
)
|
||||
g = p.add_mutually_exclusive_group(required=True)
|
||||
g.add_argument("--base", help="sha256 prefix of the node's image (`fw status`)")
|
||||
g.add_argument("--full", action="store_true", help="the full image package")
|
||||
@@ -492,7 +541,10 @@ def main() -> None:
|
||||
p.add_argument(
|
||||
"--pubkey", help="default: the release OTA key, [ota] pubkey of platformio.ini"
|
||||
)
|
||||
p.add_argument("--allowed-signers", default=str(SIGNERS))
|
||||
p.add_argument(
|
||||
"--allowed-signers",
|
||||
help="default: tools/ota_allowed_signers (--private: fw_allowed_signers)",
|
||||
)
|
||||
args = ap.parse_args()
|
||||
{
|
||||
"keygen": cmd_keygen,
|
||||
|
||||
Executable
+253
@@ -0,0 +1,253 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: MIT
|
||||
"""Push an OTA package (tools/ota_pack.py) to a node over its USB console.
|
||||
|
||||
tools/ota_push.py b1 Xiao_nrf52_repeater-ota.mcfw # transfer + verify
|
||||
tools/ota_push.py b1 Xiao_nrf52_repeater-ota.mcfw --apply # ... and apply
|
||||
tools/ota_push.py b1 pkg.mcfw --binary --drop 30 --prime old.mcfw
|
||||
|
||||
The board: a name from the local config (tools/mctools_config.py), a USB
|
||||
serial number or a port path.
|
||||
|
||||
Bench tests of the MESH protocol without a radio (--binary): the chunks go
|
||||
through `fw bin`, which feeds the very handler a binary request over the mesh
|
||||
reaches, padded to the AES block as the node sees it after decryption. Then
|
||||
the missing-units bitmap ('m') decides what to resend, as ota_mesh_push.py does.
|
||||
--drop P lose P % of the chunks of the first round on purpose
|
||||
--prime FILE first send the manifest of ANOTHER package and abort, so it
|
||||
stays in the stage flash
|
||||
--stale FILE inject the first chunks of ANOTHER package with a wrong
|
||||
session id before the real ones -- late packets of an earlier
|
||||
transfer, the cause of "wrong base image" on the bench
|
||||
|
||||
The bench path of src/helpers/FwUpdate: the same `fw` commands later travel
|
||||
over the mesh as admin CLI. Nothing here is applied without --apply, and the
|
||||
node itself refuses to apply below FW_OTA_MIN_MV (battery) or with a package
|
||||
that failed `fw verify`.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import glob
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from board_probe import _open as open_port
|
||||
from fleet import serial_for
|
||||
|
||||
CHUNK = 96 # FW_OTA_CHUNK
|
||||
BCHUNK = 96 # binary over USB: base64 of 7 + 96 B fits the 160 B console line
|
||||
UNIT = 32 # FW_OTA_UNIT
|
||||
|
||||
|
||||
def crc16(data: bytes, crc: int = 0xFFFF) -> int:
|
||||
"""Nordic crc16_compute (CCITT, init 0xFFFF) -- FwUpdate.cpp crc16_update."""
|
||||
for b in data:
|
||||
crc = ((crc >> 8) | (crc << 8)) & 0xFFFF
|
||||
crc ^= b
|
||||
crc ^= (crc & 0xFF) >> 4
|
||||
crc ^= (crc << 12) & 0xFFFF
|
||||
crc ^= ((crc & 0xFF) << 5) & 0xFFFF
|
||||
return crc
|
||||
|
||||
|
||||
def w_payload(sid: int, off: int, chunk: bytes) -> bytes:
|
||||
"""REQ_TYPE_FW_OTA 'w': [F0]['w'][sid u16][off u32][len u8][crc16][data]."""
|
||||
head = sid.to_bytes(2, "little") + off.to_bytes(4, "little") + bytes([len(chunk)])
|
||||
# the crc binds the data to this session and offset (FwUpdate::handleBinary)
|
||||
return (
|
||||
bytes([0xF0, ord("w")])
|
||||
+ head
|
||||
+ crc16(chunk, crc16(head)).to_bytes(2, "little")
|
||||
+ chunk
|
||||
)
|
||||
|
||||
|
||||
def fw_bin(s, payload: bytes) -> bytes | None:
|
||||
r = command(s, "fw bin " + base64.b64encode(payload).decode())
|
||||
if r.startswith("ERR"):
|
||||
sys.exit(f"fw bin: {r}")
|
||||
r = r.lstrip("> ").strip()
|
||||
return None if r == "-" else bytes.fromhex(r)
|
||||
|
||||
|
||||
def bin_missing(s) -> tuple[int, list[int]]:
|
||||
d = fw_bin(s, bytes([0xF0, ord("m")]))
|
||||
if not d or len(d) < 5:
|
||||
sys.exit(f"'m' reply: {d!r}")
|
||||
cnt = int.from_bytes(d[3:5], "little")
|
||||
have = (len(d) - 5) // 2 # the console prints a shortened list
|
||||
return d[0], [
|
||||
int.from_bytes(d[5 + 2 * i : 7 + 2 * i], "little")
|
||||
for i in range(min(cnt, have))
|
||||
]
|
||||
|
||||
|
||||
def send_binary(
|
||||
s, data: bytes, drop: float, sid: int, stale: bytes | None = None
|
||||
) -> None:
|
||||
if stale:
|
||||
# chunks of ANOTHER session (as if a late packet of an earlier transfer):
|
||||
# the node must drop them, or the manifest check sees the wrong package
|
||||
for off in (0, BCHUNK, 2 * BCHUNK):
|
||||
c = stale[off : off + BCHUNK]
|
||||
fw_bin(
|
||||
s,
|
||||
w_payload((sid ^ 0x5A5A), off, c),
|
||||
)
|
||||
print("sent 3 stale chunks of another session")
|
||||
todo = list(range((len(data) + BCHUNK - 1) // BCHUNK))
|
||||
for rnd in range(10):
|
||||
lost = 0
|
||||
for i in todo:
|
||||
if rnd == 0 and random.random() < drop / 100:
|
||||
lost += 1
|
||||
continue
|
||||
off = i * BCHUNK
|
||||
chunk = data[off : off + BCHUNK]
|
||||
if fw_bin(
|
||||
s,
|
||||
w_payload(sid, off, chunk),
|
||||
):
|
||||
sys.exit("a 'w' got a reply, it must not")
|
||||
if stale and rnd == 0:
|
||||
# and again AFTER the real manifest chunks: the order seen on the bench
|
||||
for off in (0, BCHUNK, 2 * BCHUNK):
|
||||
c = stale[off : off + BCHUNK]
|
||||
fw_bin(
|
||||
s,
|
||||
w_payload((sid ^ 0x5A5A), off, c),
|
||||
)
|
||||
state, miss = bin_missing(s)
|
||||
for _ in range(20):
|
||||
if state != 4:
|
||||
break
|
||||
time.sleep(0.3)
|
||||
state, miss = bin_missing(s)
|
||||
print(
|
||||
f"round {rnd + 1}: sent {len(todo) - lost}, dropped {lost}, state {state}, missing {len(miss)}+"
|
||||
)
|
||||
if state == 3:
|
||||
sys.exit(f"node: {command(s, 'fw status')}")
|
||||
if not miss:
|
||||
return
|
||||
todo = sorted({u * UNIT // BCHUNK for u in miss})
|
||||
sys.exit("binary transfer did not converge")
|
||||
|
||||
|
||||
def resolve(target: str) -> str:
|
||||
"""Same rules as serial_cli.py: fleet name, bare serial number, or a path."""
|
||||
if target.startswith("/"):
|
||||
return target
|
||||
sn = serial_for(target)
|
||||
hits = [p for p in sorted(glob.glob("/dev/serial/by-id/*")) if sn in p]
|
||||
if len(hits) != 1:
|
||||
sys.exit(f"{target!r} (sn {sn}): ports {hits}")
|
||||
return hits[0]
|
||||
|
||||
|
||||
def command(s, line: str, timeout: float = 5.0) -> str:
|
||||
"""One console command, returns the reply after '-> '."""
|
||||
s.reset_input_buffer()
|
||||
s.write(line.encode() + b"\r\n")
|
||||
buf, t = b"", time.time()
|
||||
while time.time() - t < timeout:
|
||||
d = s.read(4096)
|
||||
if d:
|
||||
buf += d
|
||||
m = re.search(rb"-> ?(.*?)\r?\n", buf)
|
||||
if m:
|
||||
return m.group(1).decode(errors="replace").strip()
|
||||
else:
|
||||
time.sleep(0.005)
|
||||
raise TimeoutError(f"no reply to {line[:40]!r}: {buf[-200:]!r}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(
|
||||
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
|
||||
)
|
||||
ap.add_argument("board", help="board name (local config), USB serial number, or port path")
|
||||
ap.add_argument("package")
|
||||
ap.add_argument("--apply", action="store_true")
|
||||
ap.add_argument("--binary", action="store_true")
|
||||
ap.add_argument("--drop", type=float, default=0.0)
|
||||
ap.add_argument("--prime")
|
||||
ap.add_argument(
|
||||
"--stale", help="package whose chunks are injected with a wrong session id"
|
||||
)
|
||||
args = ap.parse_args()
|
||||
|
||||
with open(args.package, "rb") as f:
|
||||
data = f.read()
|
||||
s = open_port(resolve(args.board))
|
||||
try:
|
||||
s.write(b"\r\n")
|
||||
time.sleep(0.4)
|
||||
print("status:", command(s, "fw status"))
|
||||
if args.prime:
|
||||
with open(args.prime, "rb") as f:
|
||||
other = f.read()
|
||||
command(s, f"fw begin {len(other)}")
|
||||
for off in range(0, 256, CHUNK):
|
||||
command(
|
||||
s,
|
||||
f"fw w {off} {base64.b64encode(other[off : off + CHUNK]).decode()}",
|
||||
)
|
||||
time.sleep(1) # let loop() check and flush it
|
||||
print(
|
||||
"primed with",
|
||||
args.prime,
|
||||
"->",
|
||||
command(s, "fw status"),
|
||||
"/",
|
||||
command(s, "fw abort"),
|
||||
)
|
||||
sid = random.randint(1, 0xFFFF)
|
||||
r = command(s, f"fw begin {len(data)} {sid}")
|
||||
print("begin:", r)
|
||||
if not r.startswith("OK"):
|
||||
sys.exit(1)
|
||||
t0 = time.time()
|
||||
n = 0 if args.binary else (len(data) + CHUNK - 1) // CHUNK
|
||||
if args.binary:
|
||||
stale = None
|
||||
if args.stale:
|
||||
with open(args.stale, "rb") as f:
|
||||
stale = f.read()
|
||||
send_binary(s, data, args.drop, sid, stale)
|
||||
for i in range(n):
|
||||
off = i * CHUNK
|
||||
b64 = base64.b64encode(data[off : off + CHUNK]).decode()
|
||||
r = command(s, f"fw w {off} {b64}")
|
||||
if r != "ok":
|
||||
sys.exit(f"chunk {off}: {r}")
|
||||
if i % 200 == 0 or i == n - 1:
|
||||
print(
|
||||
f"\r{i + 1}/{n} chunks, {time.time() - t0:.0f} s",
|
||||
end="",
|
||||
flush=True,
|
||||
)
|
||||
print()
|
||||
print("missing:", command(s, "fw missing"))
|
||||
r = command(s, "fw verify", timeout=30)
|
||||
t1 = time.time()
|
||||
while "verifying" in r and time.time() - t1 < 60: # loop() runs it
|
||||
time.sleep(0.5)
|
||||
r = command(s, "fw status")
|
||||
print("verify:", r)
|
||||
if "ready" not in r:
|
||||
sys.exit(1)
|
||||
print("bootloader:", command(s, "fw bl"))
|
||||
if args.apply:
|
||||
print("apply:", command(s, "fw apply", timeout=10))
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,75 @@
|
||||
# SPDX-License-Identifier: MIT
|
||||
# SCons script: `Import` and `env` are SCons builtins (as in build_version.py).
|
||||
# ruff: noqa: F821
|
||||
# mypy: disable-error-code="name-defined"
|
||||
"""PlatformIO extra script: run src/helpers/FwPatch.cpp from RAM (nRF52 envs).
|
||||
|
||||
[nrf52_base] extra_scripts = ... pre:tools/ota_ramlib.py
|
||||
|
||||
`fw apply` with a delta package rewrites the flash the firmware executes from,
|
||||
so the patch applier (FwPatch.cpp with the vendored detools + heatshrink inside)
|
||||
must not have a single instruction or constant table in flash. The vendored code
|
||||
stays unmodified; instead the compiled object is post-processed:
|
||||
|
||||
* every .text*/.rodata* section is renamed to .data.fwp* -- the nRF52 linker
|
||||
script collects .data* into RAM and the startup code copies it there, exactly
|
||||
like the RAMFUNC attribute in FwUpdate.cpp does for single functions;
|
||||
* memcpy/memset/memmove references become fwp_memcpy/... (RAM copies defined in
|
||||
FwPatch.cpp), because the compiler emits those calls on its own;
|
||||
* the RAM copies carry optimize("no-tree-loop-distribute-patterns") in
|
||||
FwPatch.cpp itself, otherwise GCC turns their bodies back into memcpy calls.
|
||||
|
||||
It runs as a pre-action of the LINK, on whatever FwPatch.cpp.o goes into it, and
|
||||
is idempotent. A post-action of the compile (the first version) silently did
|
||||
not run when the object was rebuilt because it was missing rather than because
|
||||
the source changed -- and the image then had its applier in flash.
|
||||
|
||||
tools/ota_check_ramfunc.sh <env> verifies the result on the linked ELF.
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
if __name__ == "__main__": # run by hand: just the help
|
||||
print(__doc__)
|
||||
sys.exit(0)
|
||||
|
||||
Import("env") # SCons builtins, see the ruff exemption at the top
|
||||
|
||||
TARGET = "helpers/FwPatch.cpp"
|
||||
REDEFINE = ("memcpy", "memset", "memmove")
|
||||
|
||||
|
||||
def _ramify(target, source, env):
|
||||
obj = target[0].get_abspath()
|
||||
objcopy = env.subst("$OBJCOPY")
|
||||
objdump = objcopy[: -len("objcopy")] + "objdump"
|
||||
heads = subprocess.run(
|
||||
[objdump, "-h", obj], check=True, capture_output=True, text=True
|
||||
).stdout
|
||||
args = []
|
||||
for line in heads.splitlines():
|
||||
parts = line.split()
|
||||
if len(parts) >= 2 and parts[0].isdigit():
|
||||
name = parts[1]
|
||||
if name.startswith((".text", ".rodata")):
|
||||
args += ["--rename-section", f"{name}=.data.fwp{name}"]
|
||||
for sym in REDEFINE:
|
||||
args += ["--redefine-sym", f"{sym}=fwp_{sym}"]
|
||||
subprocess.run([objcopy, *args, obj], check=True)
|
||||
print(
|
||||
f"ota_ramlib: {len(args) // 2 - len(REDEFINE)} sections of FwPatch.cpp moved to RAM"
|
||||
)
|
||||
|
||||
|
||||
def _prelink(target, source, env):
|
||||
hits = [
|
||||
n for n in source if n.get_path().replace("\\", "/").endswith(TARGET + ".o")
|
||||
]
|
||||
for obj in hits:
|
||||
_ramify([obj], [], env)
|
||||
if not hits:
|
||||
print("ota_ramlib: FwPatch.cpp.o not in this link (no FwUpdate here)")
|
||||
|
||||
|
||||
env.AddPreAction("$BUILD_DIR/${PROGNAME}.elf", _prelink)
|
||||
Reference in new issue
Block a user