forge: step 2 — add representative build tooling

.forge/reviews/claude-step-2.json

@@ -0,0 +1,56 @@
+{
+  "verdict": "APPROVED",
+  "blocking_issues": 0,
+  "reviewed_files": [
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PLAN.md",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/steps/step-2-plan.md",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/reviews/claude-step-2.json",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.github/workflows/firmware-build.yml",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.gitignore",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/README.md",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/scripts/build-representative.sh",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/scripts/parse-size-report.py",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/scripts/meshcore-env.sh",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/scripts/apply-patches.sh",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/vendor/MeshCore/build.sh",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/vendor/MeshCore/.github/workflows/build-companion-firmwares.yml",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/vendor/MeshCore/.github/actions/setup-build-environment/action.yml",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/vendor/MeshCore/variants/heltec_v3/platformio.ini",
+    "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/vendor/MeshCore/variants/rak4631/platformio.ini"
+  ],
+  "diff_basis": "provided_list",
+  "plan_section": "Step 2: Add representative build and size-report tooling",
+  "dimensions": {
+    "plan_alignment": {
+      "score": "PASS",
+      "notes": "The implementation matches the Step 2 goal: it builds the four planned representative environments, applies patches, captures logs/artifacts, emits size summaries, adds CI, ignores generated outputs, and documents local usage."
+    },
+    "correctness_safety": {
+      "score": "PASS",
+      "notes": "PlatformIO availability is checked with a clear local .venv fallback, compare/baseline argument conflicts are rejected in the wrapper, failed envs are preserved and reported after parsing, and missing metrics are represented explicitly rather than fabricated."
+    },
+    "code_quality": {
+      "score": "PASS",
+      "notes": "The shell and Python scripts are readable, appropriately separated by responsibility, quote paths safely, and reuse Step 1 environment definitions instead of duplicating the representative env list in CI."
+    },
+    "completeness": {
+      "score": "PASS",
+      "notes": "Acceptance criteria are satisfied based on the changed files and provided verification, including full representative builds, parser synthetic coverage, CI YAML parsing, and compare/baseline checks."
+    },
+    "patterns": {
+      "score": "PASS",
+      "notes": "The changes follow the wrapper-repository pattern, use the upstream build script rather than replacing it, and store all generated build outputs under ignored wrapper-level out/ paths."
+    },
+    "integration": {
+      "score": "PASS",
+      "notes": "The wrapper accounts for upstream build.sh clearing vendor/MeshCore/out per invocation by copying artifacts after each environment build, and the GitHub workflow uses the same local script as developers."
+    }
+  },
+  "flags": [
+    {
+      "file": "/Users/cjvana/Documents/GitHub/meshcore-bot-fw/scripts/parse-size-report.py",
+      "line": 126,
+      "note": "The generated_at timestamp means summary JSON is not byte-for-byte identical across identical inputs. This is non-blocking because ordering and schema are deterministic and generated_at is part of the step execution contract."
+    }
+  ]
+}

.forge/steps/step-2-plan.md

@@ -0,0 +1,59 @@
+# Step 2 Execution Plan: Add representative build and size-report tooling
+
+## Goal
+Create local and CI tooling that applies the MeshCore patch queue, builds the four representative companion environments, captures per-environment logs/artifacts, and emits a deterministic size JSON summary.
+
+## Current Code Observations
+- `vendor/MeshCore/build.sh` provides `build-firmware <target>` and copies artifacts to its own `out/` directory after each build.
+- `vendor/MeshCore/build.sh` removes `vendor/MeshCore/out` at startup, so wrapper tooling must copy artifacts after each environment build before starting the next one.
+- `vendor/MeshCore/build.sh` requires `FIRMWARE_VERSION` in the environment and uses the current MeshCore git SHA in artifact filenames.
+- Upstream CI installs Python 3.11 and PlatformIO with `pip install --upgrade platformio`, then runs `build.sh build-companion-firmwares`.
+- Representative envs exist in `variants/heltec_v3/platformio.ini` and `variants/rak4631/platformio.ini`; RAK4631 companion envs set `board_upload.maximum_size = 712704`.
+- The wrapper currently has no `.gitignore`, no `.github/workflows/`, and no build output tooling.
+
+## Files to Change
+- `scripts/build-representative.sh` — new wrapper build script for patch application, four-env builds, logs, copied artifacts, and size summary generation.
+- `scripts/parse-size-report.py` — new parser for PlatformIO RAM/flash output plus artifact byte sizes, with optional baseline/compare support.
+- `.github/workflows/firmware-build.yml` — new GitHub Actions workflow using the same representative build script as local development.
+- `.gitignore` — ignore generated `out/` and local PlatformIO/cache artifacts if needed.
+- `README.md` — add the local representative build command and output locations.
+
+## Ordered Implementation Checklist
+1. Add output ignores for wrapper-level generated build artifacts.
+2. Implement `scripts/parse-size-report.py` with ANSI stripping, PlatformIO RAM/flash line parsing, artifact byte discovery, deterministic JSON output, and optional `--baseline` / `--compare <json>` arguments.
+3. Implement `scripts/build-representative.sh` with `--help`, `--baseline`, `--compare <baseline-json>`, PlatformIO availability checks, patch application, per-env `build.sh build-firmware` invocation, per-env log capture, artifact copying, and parser invocation.
+4. Ensure `build-representative.sh` uses `REPRESENTATIVE_ENVS` from `scripts/meshcore-env.sh`, sets a default `FIRMWARE_VERSION` when absent, and preserves each env result rather than hiding failures.
+5. Add `.github/workflows/firmware-build.yml` to check out submodules, install Python 3.11 and PlatformIO, run `bash scripts/build-representative.sh --baseline`, and upload `out/size` plus `out/firmware` artifacts.
+6. Update `README.md` with the representative build command and generated output paths.
+7. Run shell syntax and Python compile checks, then run script help and the parser against a small synthetic log fixture without requiring PlatformIO.
+8. If PlatformIO is available locally, run the full baseline build; otherwise record that full build verification is deferred to CI or a local PlatformIO install.
+
+## Interfaces and Data Contracts
+- `scripts/build-representative.sh [--baseline|--compare <baseline-json>]` builds all envs from `REPRESENTATIVE_ENVS` and writes outputs under wrapper-level `out/`.
+- `scripts/build-representative.sh --help` prints usage and exits successfully without requiring PlatformIO.
+- `scripts/parse-size-report.py --logs <log-dir> --artifacts <artifact-dir> [--baseline|--compare <baseline-json>] [--output <json>]` emits JSON containing:
+  - `mode`
+  - `generated_at`
+  - `environments[]`
+  - per-env `env`, `ram_used`, `ram_total`, `flash_used`, `flash_total`, `artifact_bytes`, `log`, `metrics_found`, and optional `delta`
+- Missing RAM/flash metrics are represented as `null` and `metrics_found: false`; the parser must not invent successful measurements.
+- CI must use the same local scripts instead of duplicating build logic.
+
+## Verification Plan
+- Automated:
+  - `bash -n scripts/build-representative.sh scripts/meshcore-env.sh`
+  - `python3 -m py_compile scripts/parse-size-report.py`
+  - `bash scripts/build-representative.sh --help`
+  - Run `scripts/parse-size-report.py` against a synthetic log/artifact directory.
+  - If PlatformIO is installed: `bash scripts/build-representative.sh --baseline`
+- Manual:
+  - Review generated JSON shape from the synthetic parser run.
+  - Confirm workflow env list comes from the wrapper script, not a duplicated matrix.
+- Regression:
+  - `bash scripts/apply-patches.sh` still succeeds with an empty patch queue.
+  - Step 1 patch scripts remain unchanged in behavior.
+
+## Stop Conditions
+- Pause if upstream build env names differ from the planned four representative environments.
+- Pause if implementing baseline comparison would require committing generated firmware artifacts.
+- Pause if local PlatformIO installation or firmware build requires destructive cleanup outside wrapper-level `out/` or upstream `.pio` build directories.

.github/workflows/firmware-build.yml

@@ -0,0 +1,50 @@
+name: Firmware Build
+
+on:
+  pull_request:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+
+jobs:
+  representative-build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+        with:
+          submodules: recursive
+
+      - name: Cache PlatformIO
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cache/pip
+            ~/.platformio/.cache
+          key: ${{ runner.os }}-pio
+
+      - name: Install Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Install PlatformIO
+        run: python3 -m pip install --upgrade platformio
+
+      - name: Build representative firmware
+        run: bash scripts/build-representative.sh --baseline
+
+      - name: Upload size reports
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: representative-size-reports
+          path: out/size
+
+      - name: Upload firmware artifacts
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: representative-firmware
+          path: out/firmware

.gitignore

@@ -0,0 +1,6 @@
+out/
+.venv/
+__pycache__/
+*.pyc
+vendor/MeshCore/.pio/
+vendor/MeshCore/out/

README.md

@@ -24,6 +24,16 @@ Develop firmware changes in `vendor/MeshCore`, commit them in that submodule wor
 bash scripts/export-patches.sh origin/main
 ```
 
+## Representative builds
+
+Build the release-gate companion environments and write size reports:
+
+```sh
+bash scripts/build-representative.sh --baseline
+```
+
+The build writes PlatformIO logs and `summary.json` to `out/size/`, and copied firmware artifacts to `out/firmware/`.
+
 Representative companion build environments:
 
 - `Heltec_v3_companion_radio_usb`

scripts/build-representative.sh

@@ -0,0 +1,141 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/meshcore-env.sh"
+
+usage() {
+  cat <<'EOF'
+Usage: scripts/build-representative.sh [--baseline|--compare <baseline-json>]
+
+Applies the MeshCore patch queue, builds the representative companion firmware
+environments, copies firmware artifacts to out/firmware, and writes a size
+summary to out/size/summary.json.
+EOF
+}
+
+mode="summary"
+compare_file=""
+mode_selected=0
+
+while [ "$#" -gt 0 ]; do
+  case "$1" in
+    --help|-h)
+      usage
+      exit 0
+      ;;
+    --baseline)
+      if [ "$mode_selected" -eq 1 ]; then
+        echo "Choose only one of --baseline or --compare." >&2
+        exit 2
+      fi
+      mode="baseline"
+      mode_selected=1
+      shift
+      ;;
+    --compare)
+      if [ "$mode_selected" -eq 1 ]; then
+        echo "Choose only one of --baseline or --compare." >&2
+        exit 2
+      fi
+      if [ "$#" -lt 2 ]; then
+        echo "--compare requires a baseline JSON path" >&2
+        exit 2
+      fi
+      mode="compare"
+      mode_selected=1
+      compare_file="$2"
+      shift 2
+      ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 2
+      ;;
+  esac
+done
+
+if [ "$mode" = "compare" ] && [ ! -f "$compare_file" ]; then
+  echo "Baseline JSON not found: $compare_file" >&2
+  exit 1
+fi
+
+compare_tmp=""
+if [ "$mode" = "compare" ]; then
+  compare_tmp="$(mktemp)"
+  cp "$compare_file" "$compare_tmp"
+fi
+
+cleanup() {
+  if [ -n "$compare_tmp" ]; then
+    rm -f "$compare_tmp"
+  fi
+}
+trap cleanup EXIT
+
+if ! command -v pio >/dev/null 2>&1; then
+  if [ -x "${MESHCORE_FW_ROOT}/.venv/bin/pio" ]; then
+    export PATH="${MESHCORE_FW_ROOT}/.venv/bin:${PATH}"
+  else
+    echo "PlatformIO is required to build representative firmware." >&2
+    echo "Install it with pipx or a local venv, for example: python3 -m venv .venv && .venv/bin/python -m pip install --upgrade platformio" >&2
+    exit 1
+  fi
+fi
+
+if [ ! -f "${MESHCORE_DIR}/build.sh" ]; then
+  echo "MeshCore build script is missing at ${MESHCORE_DIR}/build.sh" >&2
+  exit 1
+fi
+
+"${MESHCORE_FW_ROOT}/scripts/apply-patches.sh"
+
+OUT_DIR="${MESHCORE_FW_ROOT}/out"
+LOG_DIR="${OUT_DIR}/size"
+ARTIFACT_DIR="${OUT_DIR}/firmware"
+SUMMARY_PATH="${LOG_DIR}/summary.json"
+
+rm -rf "$OUT_DIR"
+mkdir -p "$LOG_DIR" "$ARTIFACT_DIR"
+
+export FIRMWARE_VERSION="${FIRMWARE_VERSION:-local}"
+
+failed_envs=()
+for env in "${REPRESENTATIVE_ENVS[@]}"; do
+  log_path="${LOG_DIR}/${env}.log"
+  echo "Building ${env}"
+  if (cd "$MESHCORE_DIR" && /usr/bin/env bash build.sh build-firmware "$env") >"$log_path" 2>&1; then
+    if compgen -G "${MESHCORE_DIR}/out/${env}-*" >/dev/null; then
+      cp "${MESHCORE_DIR}/out/${env}-"* "$ARTIFACT_DIR"/
+    fi
+  else
+    failed_envs+=("$env")
+    echo "Build failed for ${env}; see ${log_path}" >&2
+  fi
+done
+
+parser_args=(
+  --logs "$LOG_DIR"
+  --artifacts "$ARTIFACT_DIR"
+  --output "$SUMMARY_PATH"
+)
+for env in "${REPRESENTATIVE_ENVS[@]}"; do
+  parser_args+=(--env "$env")
+done
+
+case "$mode" in
+  baseline)
+    parser_args+=(--baseline)
+    ;;
+  compare)
+    parser_args+=(--compare "$compare_tmp")
+    ;;
+esac
+
+python3 "${MESHCORE_FW_ROOT}/scripts/parse-size-report.py" "${parser_args[@]}"
+
+if [ "${#failed_envs[@]}" -gt 0 ]; then
+  echo "Representative build failed for: ${failed_envs[*]}" >&2
+  exit 1
+fi
+
+echo "Representative build complete. Summary: ${SUMMARY_PATH#${MESHCORE_FW_ROOT}/}"

scripts/parse-size-report.py

@@ -0,0 +1,151 @@
+#!/usr/bin/env python3
+import argparse
+import json
+import re
+from datetime import datetime, timezone
+from pathlib import Path
+
+ANSI_RE = re.compile(r"\x1b\[[0-9;?]*[ -/]*[@-~]")
+RAM_RE = re.compile(
+    r"RAM:\s+\[[^\]]*\]\s+\s*[0-9.]+%\s+\(used\s+([0-9]+)\s+bytes\s+from\s+([0-9]+)\s+bytes\)",
+    re.IGNORECASE,
+)
+FLASH_RE = re.compile(
+    r"Flash:\s+\[[^\]]*\]\s+\s*[0-9.]+%\s+\(used\s+([0-9]+)\s+bytes\s+from\s+([0-9]+)\s+bytes\)",
+    re.IGNORECASE,
+)
+
+
+def strip_ansi(text):
+    return ANSI_RE.sub("", text)
+
+
+def metric_from_match(match):
+    if not match:
+        return None, None
+    return int(match.group(1)), int(match.group(2))
+
+
+def parse_log(path):
+    if not path.exists():
+        return None, None, None, None, False
+
+    text = strip_ansi(path.read_text(errors="replace"))
+    ram_match = None
+    flash_match = None
+    for match in RAM_RE.finditer(text):
+        ram_match = match
+    for match in FLASH_RE.finditer(text):
+        flash_match = match
+
+    ram_used, ram_total = metric_from_match(ram_match)
+    flash_used, flash_total = metric_from_match(flash_match)
+    return ram_used, ram_total, flash_used, flash_total, bool(ram_match and flash_match)
+
+
+def artifact_sizes(artifact_dir, env):
+    sizes = {}
+    if not artifact_dir.exists():
+        return sizes
+
+    for path in sorted(artifact_dir.glob(f"{env}-*")):
+        if path.is_file():
+            sizes[path.name] = path.stat().st_size
+    return sizes
+
+
+def load_baseline(path):
+    data = json.loads(path.read_text())
+    return {entry["env"]: entry for entry in data.get("environments", [])}
+
+
+def artifact_kind(name):
+    path = Path(name)
+    suffix = path.suffix.lstrip(".") or "no_extension"
+    if path.stem.endswith("-merged") and suffix == "bin":
+        return "merged.bin"
+    return suffix
+
+
+def artifact_kind_sizes(artifacts):
+    sizes = {}
+    for name, size in artifacts.items():
+        sizes[artifact_kind(name)] = size
+    return sizes
+
+
+def add_delta(entry, baseline_by_env):
+    baseline = baseline_by_env.get(entry["env"])
+    if not baseline:
+        return
+
+    delta = {}
+    for key in ("ram_used", "flash_used"):
+        current_value = entry.get(key)
+        baseline_value = baseline.get(key)
+        if current_value is not None and baseline_value is not None:
+            delta[key] = current_value - baseline_value
+        else:
+            delta[key] = None
+
+    baseline_artifacts = artifact_kind_sizes(baseline.get("artifact_bytes", {}))
+    current_artifacts = artifact_kind_sizes(entry.get("artifact_bytes", {}))
+    artifact_delta = {}
+    for suffix, current_size in current_artifacts.items():
+        baseline_size = baseline_artifacts.get(suffix)
+        artifact_delta[suffix] = current_size - baseline_size if baseline_size is not None else None
+    delta["artifact_bytes_by_kind"] = artifact_delta
+    entry["delta"] = delta
+
+
+def build_summary(args):
+    log_dir = args.logs
+    artifact_dir = args.artifacts
+    envs = args.env or sorted(path.stem for path in log_dir.glob("*.log"))
+    baseline_by_env = load_baseline(args.compare) if args.compare else {}
+
+    entries = []
+    for env in envs:
+        log_path = log_dir / f"{env}.log"
+        ram_used, ram_total, flash_used, flash_total, metrics_found = parse_log(log_path)
+        entry = {
+            "env": env,
+            "ram_used": ram_used,
+            "ram_total": ram_total,
+            "flash_used": flash_used,
+            "flash_total": flash_total,
+            "artifact_bytes": artifact_sizes(artifact_dir, env),
+            "log": str(log_path),
+            "metrics_found": metrics_found,
+        }
+        add_delta(entry, baseline_by_env)
+        entries.append(entry)
+
+    mode = "compare" if args.compare else "baseline" if args.baseline else "summary"
+    return {
+        "mode": mode,
+        "generated_at": datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z"),
+        "environments": entries,
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Parse PlatformIO size output and firmware artifact sizes.")
+    parser.add_argument("--logs", type=Path, required=True, help="Directory containing <env>.log files")
+    parser.add_argument("--artifacts", type=Path, required=True, help="Directory containing copied firmware artifacts")
+    parser.add_argument("--env", action="append", help="Environment name to include; may be passed more than once")
+    parser.add_argument("--baseline", action="store_true", help="Mark output as a baseline summary")
+    parser.add_argument("--compare", type=Path, help="Baseline JSON to compare against")
+    parser.add_argument("--output", type=Path, help="Write JSON summary to this path")
+    args = parser.parse_args()
+
+    summary = build_summary(args)
+    text = json.dumps(summary, indent=2, sort_keys=True)
+    if args.output:
+        args.output.parent.mkdir(parents=True, exist_ok=True)
+        args.output.write_text(text + "\n")
+    print(text)
+
+
+if __name__ == "__main__":
+    main()