ua1zbe/meshcore-bot-firmware
1
0
Fork 0
mirror of https://github.com/Colorado-Mesh/meshcore-bot-firmware.git synced 2026-08-11 16:20:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

forge: step 1 — bootstrap wrapper workflow

Browse Source
This commit is contained in:
cj-vana
2026-05-14 12:08:59 -06:00
parent b28469d1d9
commit 346ca7cb93
19 changed files with 2192 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.forge/.base-ref Normal file
View File

@@ -0,0 +1 @@
b28469d1d9c07eda4c725d9d703b99724f86f380

713
.forge/PLAN.md Normal file
View File

@@ -0,0 +1,713 @@
# Forge Implementation Plan
## Overview
Build a Colorado Mesh firmware-only bot as a wrapper repository around upstream `meshcore-dev/MeshCore`, pinned as a submodule and modified through a deterministic patch queue. The firmware bot will live inside MeshCore companion firmware, handle lightweight fun + utility commands directly on-device, keep normal bot traffic off Public, allow DMs plus `#bot`/`#testing`, forward `#emergency` to Public, coordinate/suppress normal duplicate bot replies with passive listen-before-answer, expose runtime CLI/config controls, harden production key handling, measure Heltec v3 and RAK4631 build sizes, and flash the plugged-in Heltec v3 after verification passes.
## Technical Decisions
- **Firmware-first architecture:** Implement the bot as embedded C++ in MeshCore companion firmware; use `meshcore-bot` and Colorado community bot behavior as references only. Trace: ITEM-stack-1, ITEM-stack-5, ITEM-stack-6, ITEM-architecture-1, ITEM-prior-art-3.
- **Repository shape:** Keep upstream MeshCore as a pinned submodule under `vendor/MeshCore`; keep Colorado modifications as `colorado/` overlay files plus `patches/meshcore/*.patch` and scripts that apply/export patches deterministically. Trace: ITEM-stack-2, ITEM-stack-3, ITEM-architecture-14.
- **Runtime placement:** Integrate at `examples/companion_radio/MyMesh` callback/loop level and do not touch MeshCore routing, dispatcher, ACKs, packet duplicate tables, or lower-layer `txdelay`/`rxdelay` behavior. Trace: ITEM-architecture-1, ITEM-architecture-12, ITEM-pitfalls-12.
- **Traffic policy:** Normal bot responses are allowed only in DMs, `#bot`, and `#testing`; Public channel command traffic is ignored silently; `#emergency` is a special route-to-Public path. Trace: ITEM-architecture-3, ITEM-prior-art-12, PROJECT.md.
- **Emergency policy:** `#emergency` messages must be posted to Public as `EMERGENCY MESSAGE FROM <user>` followed by the original text, may be multipart, and must never be suppressed; loop prevention and rate limits bound amplification. Trace: ITEM-architecture-4, ITEM-pitfalls-8, PROJECT.md.
- **Command scope:** Implement more bot parity than the compact minimum, but only firmware-feasible fun + utility commands: no HTTP/TLS/API feeds, SQLite/history DB, Discord/web viewer, dynamic plugins, or large text catalogs until RAK4631 size evidence proves headroom. Trace: ITEM-stack-6, ITEM-prior-art-3, ITEM-prior-art-11, ITEM-pitfalls-2.
- **Coordinator:** Use passive listen-before-answer for normal traffic with semantic fingerprints, known-bot trust, bounded pending/recent tables, and no explicit on-air claim frames in v1. Trace: ITEM-architecture-5, ITEM-architecture-6, ITEM-architecture-7, ITEM-pitfalls-9, ITEM-pitfalls-11.
- **Runtime config:** Add compact bot runtime CLI/config controls in Phase 1, persisted in a separate versioned bot prefs file rather than changing `NodePrefs` binary layout. Trace: ITEM-architecture-9, ITEM-pitfalls-16.
- **Resource budget:** RAK4631 USB/BLE is the hard release gate. Aim for <=25 KB incremental app flash, <=2 KB static RAM initially, hard-review above 40-60 KB flash or 4-8 KB RAM, and keep persistent bot config under 4 KB. Trace: ITEM-stack-10, ITEM-stack-11, ITEM-stack-12, ITEM-pitfalls-1.
- **Build and hardware:** CI and local scripts must build Heltec v3 USB/BLE and RAK4631 USB/BLE with size reports. The plugged-in Heltec v3 is the first hardware smoke target after builds and reviews pass. Trace: ITEM-stack-8, ITEM-stack-13, PROJECT.md.
- **Production key hardening:** Production bot firmware disables upstream private key import/export build flags by default; a separate provisioning/dev path can be added later if needed. Trace: ITEM-pitfalls-18, PROJECT.md.
## Implementation Steps
### Step 1: Bootstrap wrapper repository, upstream submodule, and patch workflow
**Goal:** Create a reproducible Colorado Mesh firmware wrapper that pins upstream MeshCore and can apply/export Colorado bot patches without relying on untracked submodule edits.
**Why now:** All later source changes must have a stable upstream base and a clean path for CI and review.
**Dependencies:** Empty local repo initialized by Forge; git available locally; upstream MeshCore reachable; user selected submodule strategy.
**Files:**
- `.gitmodules`
- `vendor/MeshCore` submodule pointer
- `scripts/apply-patches.sh`
- `scripts/export-patches.sh`
- `scripts/meshcore-env.sh`
- `patches/meshcore/.gitkeep`
- `colorado/README.md`
- `README.md` or `docs/development.md` only if needed for operator build instructions
**Existing code to inspect first:**
- `/Users/cjvana/Documents/GitHub/MeshCore/build.sh`
- `/Users/cjvana/Documents/GitHub/MeshCore/platformio.ini`
- `/Users/cjvana/Documents/GitHub/MeshCore/variants/heltec_v3/platformio.ini`
- `/Users/cjvana/Documents/GitHub/MeshCore/variants/rak4631/platformio.ini`
**Implementation plan:**
1. Add `vendor/MeshCore` as a submodule from `https://github.com/meshcore-dev/MeshCore.git` and pin the exact upstream commit used for this run.
2. Create `patches/meshcore/` as the source-of-truth patch series directory, initially empty except for `.gitkeep`.
3. Write `scripts/meshcore-env.sh` to define `MESHCORE_DIR`, representative environment names, output paths, and common helper variables without modifying shell global state.
4. Write `scripts/apply-patches.sh` to verify the submodule exists, fail if it has unexpected uncommitted changes, apply `patches/meshcore/*.patch` in sorted order when present, and print the pinned upstream SHA.
5. Write `scripts/export-patches.sh` to export any committed Colorado changes from the submodule or a temporary working branch back to `patches/meshcore/` in deterministic order.
6. Add a small development note explaining that implementation edits happen in `vendor/MeshCore` for buildability, then patches are exported before review/commit.
7. Verify the script behavior with an empty patch queue and no source edits.
**Contracts and interfaces:**
- `scripts/apply-patches.sh` exits non-zero if patches fail or the submodule is missing.
- `scripts/export-patches.sh` never silently overwrites patches without regenerating the full ordered patch queue.
- Representative env names are: `Heltec_v3_companion_radio_usb`, `Heltec_v3_companion_radio_ble`, `RAK_4631_companion_radio_usb`, `RAK_4631_companion_radio_ble`.
**State/data changes:** Git submodule metadata and wrapper scripts only; no firmware behavior changes.
**Edge cases:** Missing submodule init, stale local submodule changes, empty patch directory, upstream branch default not named `main`, patch filenames with spaces.
**Acceptance criteria:**
- `git submodule status` shows a pinned MeshCore commit.
- `scripts/apply-patches.sh` succeeds with no patches.
- Wrapper scripts do not modify firmware files when the patch queue is empty.
**Verification commands:**
- `git submodule status`
- `bash scripts/apply-patches.sh`
- `bash -n scripts/apply-patches.sh scripts/export-patches.sh scripts/meshcore-env.sh`
**Manual validation:** Inspect the printed upstream SHA and confirm it matches the submodule pointer in git status.
**Risks:**
- Patch workflow drift can make builds non-reproducible. Mitigated by sorted patch application and failing on dirty submodule state. Trace: ITEM-stack-2, ITEM-architecture-14.
- A stale local MeshCore checkout may differ from upstream; the submodule pin becomes the authoritative source. Trace: ITEM-stack-3.
**Out of scope for this step:** Bot code, CI, PlatformIO installation, firmware build fixes.
### Step 2: Add representative build and size-report tooling
**Goal:** Create local/CI build tooling that applies patches, builds the four representative companion environments, captures artifact sizes, and reports flash/RAM usage so storage estimates can be replaced by measurements.
**Why now:** The user explicitly asked how much storage the bot firmware will take; build reporting must exist before feature growth.
**Dependencies:** Step 1 scripts; upstream MeshCore `build.sh`; PlatformIO may need installation in CI/local environment.
**Files:**
- `scripts/build-representative.sh`
- `scripts/parse-size-report.py`
- `.github/workflows/firmware-build.yml`
- `.gitignore` entries for build outputs if needed
- `colorado/size-baseline/README.md` or generated JSON baseline location
**Existing code to inspect first:**
- `vendor/MeshCore/build.sh`
- `vendor/MeshCore/.github/workflows/build-companion-firmwares.yml`
- `vendor/MeshCore/.github/actions/setup-build-environment/action.yml`
- `vendor/MeshCore/variants/heltec_v3/platformio.ini`
- `vendor/MeshCore/variants/rak4631/platformio.ini`
**Implementation plan:**
1. Write `scripts/build-representative.sh` to run `scripts/apply-patches.sh`, then invoke upstream `build.sh build-firmware` for each representative env from `vendor/MeshCore`.
2. Capture stdout/stderr per environment into `out/size/<env>.log` and copy generated `.bin`/`.uf2` artifacts into wrapper-level `out/firmware/` when present.
3. Write `scripts/parse-size-report.py` to parse PlatformIO RAM/flash usage lines and artifact byte sizes into a JSON summary.
4. Add a baseline mode that records current no-bot submodule size outputs when builds first succeed, and a compare mode that reports deltas after bot patches exist.
5. Add a GitHub Actions workflow that checks out submodules, installs PlatformIO using upstream setup guidance, runs the representative build script, uploads logs/artifacts, and prints the JSON summary.
6. Make the initial workflow warn/report rather than enforce deltas until a measured baseline exists.
7. Document the exact local command to run before hardware flashing.
**Contracts and interfaces:**
- `scripts/build-representative.sh [--baseline|--compare]` builds all four representative envs.
- `scripts/parse-size-report.py <log-dir> <artifact-dir>` emits deterministic JSON with `env`, `ram_used`, `ram_total`, `flash_used`, `flash_total`, `artifact_bytes`, and optional delta fields.
- CI uses the same scripts as local development.
**State/data changes:** Build artifacts under ignored `out/`; optional committed baseline JSON only after first successful measured baseline.
**Edge cases:** PlatformIO missing locally, upstream `build.sh` output format changes, no artifact produced for failed build, UF2 size not equal to raw app flash, logs containing ANSI color codes.
**Acceptance criteria:**
- CI workflow syntax is valid.
- Local scripts fail clearly if PlatformIO is unavailable.
- Size parser handles missing metrics without pretending the build passed.
- Representative env list matches the plan and research.
**Verification commands:**
- `bash -n scripts/build-representative.sh`
- `python3 -m py_compile scripts/parse-size-report.py`
- `bash scripts/build-representative.sh --help`
- If PlatformIO is installed by this step or already available: `bash scripts/build-representative.sh --baseline`
**Manual validation:** Review `out/size/*.log` and JSON summary after the first successful build.
**Risks:**
- RAK4631 BLE is the limiting target and may fail before bot code if local toolchain differs. Mitigate by using upstream CI setup and preserving dependency pins. Trace: ITEM-stack-4, ITEM-stack-10, ITEM-pitfalls-1.
- GitHub asset/UF2 sizes are not exact app flash, so parser must prefer PlatformIO section usage over artifact byte size. Trace: ITEM-architecture-11, ITEM-prior-art-11.
**Out of scope for this step:** Enforcing size limits, bot behavior, flashing hardware.
### Step 3: Introduce the firmware bot core module and host-side unit harness
**Goal:** Add platform-neutral firmware bot C++ source files with bounded data structures plus a host-side unit harness for parser, policy, fingerprints, and response formatting before wiring into `MyMesh`.
**Why now:** Parser and policy bugs are high-risk; testing core logic outside hardware shortens iteration.
**Dependencies:** Step 1 patch workflow; Step 2 build script for later firmware builds.
**Files:**
- `vendor/MeshCore/examples/companion_radio/FirmwareBot.h`
- `vendor/MeshCore/examples/companion_radio/FirmwareBot.cpp`
- `vendor/MeshCore/examples/companion_radio/BotPolicy.h`
- `vendor/MeshCore/examples/companion_radio/BotPolicy.cpp`
- `vendor/MeshCore/examples/companion_radio/BotTypes.h`
- `vendor/MeshCore/examples/companion_radio/BotStats.h`
- `tests/firmware_bot/` host harness files
- `patches/meshcore/0001-companion-firmware-bot-core.patch` after export
**Existing code to inspect first:**
- `vendor/MeshCore/src/MeshCore.h` for message size constants
- `vendor/MeshCore/src/helpers/BaseChatMesh.h`
- `vendor/MeshCore/src/helpers/BaseChatMesh.cpp`
- `vendor/MeshCore/examples/companion_radio/MyMesh.h`
- `vendor/MeshCore/examples/companion_radio/MyMesh.cpp`
- `vendor/MeshCore/README.md` for allocation guidance
**Implementation plan:**
1. Define `BotMessage`, `BotResponse`, `BotChannelKind`, `BotCommand`, `BotFingerprint`, `BotPrefs`, and `BotStats` structs with fixed-size fields and explicit size comments verified by static assertions.
2. Implement bounded input normalization over `(const char*, size_t)` with no reliance on untrusted `strlen` and no hot-path heap allocation.
3. Implement exact channel policy helpers for DM, `#bot`, `#testing`, `#emergency`, and Public ignore behavior.
4. Implement a 64-bit semantic fingerprint helper over stable application fields using an upstream-available hash primitive or a small deterministic local hash if SHA-256 is not already convenient in this layer.
5. Implement response buffer helpers that write into caller-provided fixed buffers and return truncation status.
6. Add a host-side C++/Python-driven unit harness that compiles the bot core with stubs and tests max-length messages, UTF-8/control chars, empty commands, colons, Public ignore, allowed channels, and fingerprint stability.
7. Export the resulting submodule changes into the first patch file.
**Contracts and interfaces:**
- Bot core exposes no Arduino `String`, `malloc`, `new`, `std::vector`, `std::map`, or JSON dependency in receive/parse/schedule paths.
- `BotPolicy::classifyChannel()` returns explicit `allow_normal`, `emergency_forward`, or `ignore` decisions.
- All output APIs require destination buffer and length.
**State/data changes:** No persistent device state yet; only in-memory structs and tests.
**Edge cases:** Embedded NULs, non-ASCII characters, messages with no command prefix, command aliases with trailing punctuation, channel names with missing leading `#`, channel names like `#botnet`, long sender names.
**Acceptance criteria:**
- Host tests pass for parser/policy/fingerprint behavior.
- Bot files compile in host harness without Arduino-specific includes except where guarded.
- Static assertions keep key structs within the planned RAM budget.
- Patch exports cleanly and reapplies from a clean submodule.
**Verification commands:**
- `bash scripts/apply-patches.sh`
- Host test command selected during implementation, e.g. `python3 tests/firmware_bot/run_tests.py`
- `grep -R "String\|malloc\|new \|std::vector\|std::map" vendor/MeshCore/examples/companion_radio/FirmwareBot* vendor/MeshCore/examples/companion_radio/Bot*` should only show allowed false positives if any.
- `bash scripts/export-patches.sh`
- `bash scripts/apply-patches.sh`
**Manual validation:** Read generated patch and confirm it only adds bot core/test support without touching routing code.
**Risks:**
- Unsafe parsing can cause buffer errors or command misfires. Mitigated by `(ptr,len)` tests and bounded output. Trace: ITEM-pitfalls-6.
- Heap fragmentation from convenience APIs can destabilize long-running nodes. Mitigated by grep/review and fixed buffers. Trace: ITEM-pitfalls-3.
**Out of scope for this step:** `MyMesh` integration, actual command handlers beyond parser stubs, persistence, emergency sending.
### Step 4: Wire BotRuntime into MyMesh callbacks and enforce channel policy
**Goal:** Instantiate the bot runtime inside companion `MyMesh`, feed it DMs/channel messages, tick it from the main loop, and enforce silent Public ignore plus allowed DM/`#bot`/`#testing` routing without sending real bot responses yet.
**Why now:** Policy must be structurally correct before command execution or emergency forwarding can create on-air traffic.
**Dependencies:** Step 3 bot core.
**Files:**
- `vendor/MeshCore/examples/companion_radio/MyMesh.h`
- `vendor/MeshCore/examples/companion_radio/MyMesh.cpp`
- `vendor/MeshCore/examples/companion_radio/FirmwareBot.*`
- `patches/meshcore/0002-wire-firmware-bot-runtime.patch`
- Host tests updated for adapter behavior if feasible
**Existing code to inspect first:**
- `vendor/MeshCore/examples/companion_radio/MyMesh.h:124` and nearby receive declarations
- `vendor/MeshCore/examples/companion_radio/MyMesh.cpp` `onMessageRecv`, `onChannelMessageRecv`, constructor, `loop`, `handleCmdFrame`
- `vendor/MeshCore/src/helpers/BaseChatMesh.cpp` group message parsing and send behavior
- `vendor/MeshCore/src/helpers/ChannelDetails.h`
**Implementation plan:**
1. Add `FirmwareBot` member storage to `MyMesh` behind `CMESH_BOT_ENABLED` compile flag so stock builds can compile it out if needed.
2. Initialize bot runtime with node name, RNG/time accessors, channel resolver, DataStore/storage pointers where needed, and a send adapter that is initially disabled or dry-run for normal commands.
3. In `onMessageRecv`, convert private message inputs into `BotMessage` with contact/key/timestamp metadata after existing UI/app notification behavior remains intact.
4. In `onChannelMessageRecv`, resolve channel name/index and pass only application-level channel messages into `BotPolicy`, preserving existing companion notifications.
5. Add a `MyMesh::loop()` tick call that lets the bot runtime process timers without blocking.
6. Add debug/stat counters for observed/ignored/eligible/emergency-classified messages without sending normal responses.
7. Verify Public channel normal commands are ignored silently by policy before any response path is enabled.
**Contracts and interfaces:**
- Existing companion app/BLE/serial callbacks still receive messages as before.
- Bot runtime does not allocate MeshCore packets during receive callbacks.
- Bot runtime tick is non-blocking and does not call `delay()`.
**State/data changes:** In-memory counters only; no persistent config yet unless default compile-time prefs are introduced.
**Edge cases:** Missing `#bot` or `#testing` channels, channel index changes, DMs from unknown contacts, signed message variants, channel messages with spoofed sender prefixes.
**Acceptance criteria:**
- Firmware still builds for representative envs or fails only because PlatformIO is unavailable locally.
- Host/unit tests show Public normal traffic is ignored, `#bot`/`#testing` are eligible, DMs are eligible, and `#emergency` is diverted.
- No routing/Dispatcher/Mesh core files are modified.
**Verification commands:**
- `python3 tests/firmware_bot/run_tests.py`
- `grep -R "delay(" vendor/MeshCore/examples/companion_radio/FirmwareBot* vendor/MeshCore/examples/companion_radio/Bot* vendor/MeshCore/examples/companion_radio/MyMesh.cpp`
- If PlatformIO available: `bash scripts/build-representative.sh --compare`
- `bash scripts/export-patches.sh && bash scripts/apply-patches.sh`
**Manual validation:** Inspect the `MyMesh` diff and confirm hooks are narrow and existing companion notifications remain.
**Risks:**
- Blocking the companion loop can break BLE/serial/radio responsiveness. Mitigated by FSM tick and no delays. Trace: ITEM-pitfalls-12.
- Wrong channel routing can make the bot talk on Public. Mitigated by central policy before command execution. Trace: ITEM-architecture-3, ITEM-pitfalls-7.
**Out of scope for this step:** Actual command response text, emergency Public sending, persistent runtime config, suppression across bots.
### Step 5: Add firmware command executor for fun + utility command parity
**Goal:** Implement firmware-feasible fun + utility commands directly in C++ with bounded single-packet responses and output caps.
**Why now:** With policy enforced, normal bot behavior can be added safely before duplicate coordination.
**Dependencies:** Step 4 message routing; command scope decisions.
**Files:**
- `vendor/MeshCore/examples/companion_radio/BotCommands.h`
- `vendor/MeshCore/examples/companion_radio/BotCommands.cpp`
- `vendor/MeshCore/examples/companion_radio/FirmwareBot.*`
- `tests/firmware_bot/commands.*` or equivalent fixtures
- `patches/meshcore/0003-add-firmware-bot-commands.patch`
**Existing code to inspect first:**
- `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/commands/*` for command names and style
- `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/command_manager.py`
- `vendor/MeshCore/src/helpers/BaseChatMesh.h` text constants
- `vendor/MeshCore/docs/payloads.md`
**Implementation plan:**
1. Implement a fixed command table for `ping`, `test`, `hello`, `help`, `cmd`, `about`, `dice`, `roll`, `status`, `channels`, and a minimal `path`/heard diagnostic if the needed metadata is already available in `BotMessage`.
2. Add aliases that are inexpensive and useful from existing bot behavior, avoiding any API/network/database-backed commands.
3. Keep help/cmd output compact and possibly split by category only if the user explicitly requests; default response must fit within MeshCore text constraints.
4. Implement dice/roll parsing with bounded numeric ranges to prevent long output or overflow.
5. Implement status using existing battery/storage/stat values available through `MyMesh`/DataStore, not new filesystem scans or heavy telemetry.
6. Add per-command cooldown/rate-limit fields in runtime state, with low-risk defaults and no persistent history.
7. Add golden tests for each command, output length caps, disallowed Public behavior, and malformed arguments.
8. Wire `MyMesh` send adapter to transmit approved normal responses to DMs or allowed group channels.
**Contracts and interfaces:**
- Every command writes into a fixed response buffer and returns `handled`, `not_handled`, or `error_short_response`.
- Normal group replies use existing `sendGroupMessage(timestamp, channel, node_name, text, text_len)`.
- DM replies use existing `sendMessage()` with expected ACK/timeout handling.
- No command uses network, filesystem logs, dynamic plugins, SQLite, JSON, or heap containers.
**State/data changes:** Runtime cooldown counters in RAM; no persistent command history.
**Edge cases:** `roll 0d0`, huge dice counts, negative sides, empty help, unknown commands, long user args, message exactly at max length, output truncation.
**Acceptance criteria:**
- Commands respond in DMs, `#bot`, and `#testing` only.
- Public channel commands produce no reply.
- All outputs fit configured safe text limits.
- Firmware builds remain within measured or estimated size budget.
**Verification commands:**
- `python3 tests/firmware_bot/run_tests.py`
- `grep -R "http\|sqlite\|requests\|JSON\|String\|std::vector\|malloc\|new " vendor/MeshCore/examples/companion_radio/Bot* vendor/MeshCore/examples/companion_radio/FirmwareBot*`
- If PlatformIO available: `bash scripts/build-representative.sh --compare`
**Manual validation:** Review command output copy for LoRa-appropriate brevity and match it against `meshcore-bot` behavior where feasible.
**Risks:**
- Python bot feature creep can exceed firmware budgets. Mitigated by excluding API/database/dynamic commands. Trace: ITEM-pitfalls-2, ITEM-prior-art-3.
- Text/frame limits can truncate replies. Mitigated by explicit output caps and tests. Trace: ITEM-pitfalls-5.
**Out of scope for this step:** Emergency forwarding, duplicate suppression, runtime CLI persistence, key import/export hardening.
### Step 6: Implement #emergency to Public forwarding with bounded multipart and loop prevention
**Goal:** Implement the required emergency bridge from `#emergency` to Public using the exact prefix, preserving the original text across bounded multipart messages, never suppressing emergency forwarding, and preventing recursive loops/amplification.
**Why now:** Emergency behavior is distinct from normal commands and should be isolated before normal duplicate suppression is added.
**Dependencies:** Step 4 channel routing; Step 5 send adapter.
**Files:**
- `vendor/MeshCore/examples/companion_radio/EmergencyForwarder.h`
- `vendor/MeshCore/examples/companion_radio/EmergencyForwarder.cpp`
- `vendor/MeshCore/examples/companion_radio/FirmwareBot.*`
- `vendor/MeshCore/examples/companion_radio/MyMesh.*` if Public channel lookup/send adapter needs additions
- `tests/firmware_bot/emergency.*`
- `patches/meshcore/0004-add-emergency-forwarder.patch`
**Existing code to inspect first:**
- `vendor/MeshCore/src/helpers/BaseChatMesh.cpp` group send prefixing and max text behavior
- `vendor/MeshCore/examples/companion_radio/MyMesh.cpp` channel lookup and send frame logic
- `vendor/MeshCore/docs/payloads.md`
- `/Users/cjvana/Documents/GitHub/meshcore-community-bot/community/message_interceptor.py` for existing emergency intent
**Implementation plan:**
1. Implement a dedicated `EmergencyForwarder` path triggered only by exact `#emergency` channel policy classification.
2. Format Public output beginning with `EMERGENCY MESSAGE FROM <user>` followed by the original text, preserving exact user text as much as MeshCore limits allow.
3. Support bounded multipart output with a compile-time max part count and per-part length caps; include part numbering or continuation only if needed and short enough.
4. Add loop prevention: ignore bot-originated Public messages beginning with the emergency prefix, ignore messages already forwarded by this node, and do not reprocess Public as an emergency source.
5. Add rate limiting and queue checks so a malicious or accidental flood cannot monopolize packet pools; failed forwards increment stats.
6. Do not apply normal duplicate suppression to emergency forwarding; multiple bots may forward the same emergency if hidden nodes exist.
7. Add tests for short, exactly-fit, long, very long, prefix-spoofed, Public-origin, and repeated emergency messages.
**Contracts and interfaces:**
- Emergency source channel is exact `#emergency`; destination channel is exact `Public`.
- Emergency forwarding never waits for normal coordinator suppression.
- Emergency formatter returns bounded parts and never emits unbounded multipart output.
- Public normal bot commands still remain silent.
**State/data changes:** RAM-only emergency rate-limit/loop-prevention state; no persistent emergency history.
**Edge cases:** Missing Public channel, missing #emergency channel, long sender names, long original text, text already starting with emergency prefix, multiple bots forwarding concurrently, packet pool unavailable.
**Acceptance criteria:**
- `#emergency` input produces Public message parts with required prefix and original content.
- Public input with the same text does not re-forward.
- Emergency forwarding is never suppressed by normal known-bot response suppression.
- Multipart is bounded and rate-limited.
**Verification commands:**
- `python3 tests/firmware_bot/run_tests.py`
- If PlatformIO available: `bash scripts/build-representative.sh --compare`
- Manual grep confirming `EmergencyForwarder` does not call suppression APIs before sending.
**Manual validation:** Review sample emergency outputs for clarity, truncation behavior, and Public channel safety.
**Risks:**
- Emergency loops/amplification can flood Public. Mitigated by loop prevention and bounded multipart/rate limits while honoring the never-suppress decision. Trace: ITEM-pitfalls-8, PROJECT.md.
- Long emergency messages can consume airtime. Mitigated by max part count and concise formatting. Trace: ITEM-pitfalls-5.
**Out of scope for this step:** Normal command duplicate suppression, signed emergency authentication, Discord/webhook forwarding.
### Step 7: Add passive response coordinator and known-bot trust registry for normal traffic
**Goal:** Reduce duplicate normal bot replies with fixed-size passive listen-before-answer suppression that trusts known bot identities and never applies to emergency forwarding.
**Why now:** Command behavior exists; now it can be delayed/canceled safely for normal bot traffic.
**Dependencies:** Step 5 normal command responses; Step 6 emergency exemption.
**Files:**
- `vendor/MeshCore/examples/companion_radio/ResponseCoordinator.h`
- `vendor/MeshCore/examples/companion_radio/ResponseCoordinator.cpp`
- `vendor/MeshCore/examples/companion_radio/KnownBotRegistry.h`
- `vendor/MeshCore/examples/companion_radio/KnownBotRegistry.cpp`
- `vendor/MeshCore/examples/companion_radio/FirmwareBot.*`
- `tests/firmware_bot/coordinator.*`
- `patches/meshcore/0005-add-response-coordinator.patch`
**Existing code to inspect first:**
- `vendor/MeshCore/src/helpers/SimpleMeshTables.h`
- `vendor/MeshCore/src/Packet.h`
- `vendor/MeshCore/examples/companion_radio/MyMesh.cpp` message metadata and contact lookup
- `/Users/cjvana/Documents/GitHub/meshcore-community-bot/community/message_interceptor.py`
**Implementation plan:**
1. Implement fixed pending/recent fingerprint tables sized for RAK4631: initial target 8-12 pending and 16-24 recent entries.
2. Define normal-response FSM states: observed, eligible, pending self, sent self, suppressed by known bot, expired, failed.
3. Compute bot-specific response delay from channel priority, command priority, configured tier bias, queue health, deterministic bot-key/fingerprint tie-breaker, and bounded jitter; do not use MeshCore `txdelay`/`rxdelay` values directly.
4. Delay normal command send by scheduling `BotResponse` metadata only; allocate MeshCore packets only when the timer wins.
5. Detect known bot responses using key-backed identity when available; weak group-name hints may be recorded for stats but must not suppress safety-critical/emergency paths.
6. Add `KnownBotRegistry` fixed slots for public keys/labels/capability flags, initially populated by runtime config/CLI later or compile-time defaults for tests.
7. Add tests for timer ordering, cancellation, TTL expiry, table full, hidden-node duplicate tolerance, Public ignore, emergency bypass, and known/unknown bot response handling.
**Contracts and interfaces:**
- Coordinator applies only to normal DM/#bot/#testing responses.
- Emergency forwarding bypasses coordinator completely.
- Known-bot authoritative trust requires public key identity where MeshCore exposes it; weak text hints cannot suppress emergency and are disabled or low-risk only by default.
- No explicit claim frames are transmitted in v1.
**State/data changes:** RAM-only pending/recent suppression state and known-bot registry loaded from config when Step 8 lands.
**Edge cases:** Table full, timer wraparound, send failure, repeated command by same user, two users sending same command, unknown spoofed bot text, known bot response after local send, hidden nodes not hearing each other.
**Acceptance criteria:**
- Normal duplicate command tests show later local response is canceled when a trusted known bot response is observed first.
- Unknown/spoofed bot text does not authoritatively suppress.
- Emergency forwarding tests prove no suppression path is used.
- No MeshCore routing or packet duplicate logic is modified.
**Verification commands:**
- `python3 tests/firmware_bot/run_tests.py`
- `grep -R "txdelay\|rxdelay\|direct.txdelay" vendor/MeshCore/examples/companion_radio/*Bot* vendor/MeshCore/examples/companion_radio/ResponseCoordinator*` should show no direct dependency except comments/tests if any.
- If PlatformIO available: `bash scripts/build-representative.sh --compare`
**Manual validation:** Review FSM transition table in tests and confirm stats distinguish suppressed, sent, expired, and failed sends.
**Risks:**
- Passive suppression cannot guarantee exactly-once under hidden nodes. Mitigated by telemetry and acceptance of rare duplicates. Trace: ITEM-pitfalls-9.
- Spoofed group text can silence real bots if trusted. Mitigated by known-key trust and emergency exemption. Trace: ITEM-pitfalls-11, ITEM-architecture-7.
**Out of scope for this step:** Persistent config, CLI commands, explicit claim frames, cryptographic claim signing.
### Step 8: Add BotPrefs persistence and runtime CLI/config commands
**Goal:** Persist small bot settings separately from `NodePrefs` and expose runtime CLI/config controls for enablement, channels, tier/delay, known bots, command toggles, and stats.
**Why now:** The user wants runtime CLI config in Phase 1, and known-bot trust needs field configuration.
**Dependencies:** Steps 4-7 runtime pieces.
**Files:**
- `vendor/MeshCore/examples/companion_radio/BotPrefs.h`
- `vendor/MeshCore/examples/companion_radio/BotPrefs.cpp`
- `vendor/MeshCore/examples/companion_radio/DataStore.h`
- `vendor/MeshCore/examples/companion_radio/DataStore.cpp`
- `vendor/MeshCore/examples/companion_radio/MyMesh.h`
- `vendor/MeshCore/examples/companion_radio/MyMesh.cpp`
- `vendor/MeshCore/src/helpers/CommonCLI.*` only if rescue CLI integration belongs there after inspection
- `tests/firmware_bot/prefs.*`
- `patches/meshcore/0006-add-bot-prefs-and-cli.patch`
**Existing code to inspect first:**
- `vendor/MeshCore/examples/companion_radio/DataStore.cpp`
- `vendor/MeshCore/examples/companion_radio/DataStore.h`
- `vendor/MeshCore/examples/companion_radio/NodePrefs.h`
- `vendor/MeshCore/examples/companion_radio/MyMesh.cpp` `handleCmdFrame`, `enterCLIRescue`, `checkCLIRescueCmd`
- `vendor/MeshCore/src/helpers/CommonCLI.cpp`
- `vendor/MeshCore/docs/companion_protocol.md` if present, plus official docs for frame conventions
**Implementation plan:**
1. Define a compact versioned `BotPrefsV1` blob with magic, version, length, CRC/checksum, enable flag, tier/delay/jitter, cooldowns, channel names or IDs, emergency mapping, known-bot slots, command bitset, and production hardening flags.
2. Add DataStore load/save helpers for a separate bot prefs file such as `/bot_prefs_v1`, without modifying `NodePrefs` persisted offsets.
3. Default safely on missing/corrupt/incompatible bot prefs while preserving the user's desired behavior once configured: enabled bot runtime, Public ignored, `#bot`/`#testing` allowed, `#emergency` to Public.
4. Add rescue CLI commands or companion command-frame handlers for `bot enable|disable`, `bot channels`, `bot tier`, `bot delay`, `bot known add/remove/list`, `bot commands`, and `bot stats`.
5. Add minimal binary protocol/query support only if existing companion command-frame handling can safely allocate a new range; otherwise keep CLI-only for v1 and record protocol config as deferred.
6. Persist known bot public keys as full keys where available, with labels/capability flags bounded to fixed slots.
7. Add tests for serialization round-trip, corrupt CRC, version mismatch, full known-bot table, invalid channel names, invalid delay values, and stats output.
**Contracts and interfaces:**
- Bot prefs are independent from `NodePrefs`; no existing `NodePrefs` offsets change.
- CLI/config inputs are length-checked and reject malformed hex keys/delays/channels.
- Runtime config changes take effect without reboot where safe; persistence errors report via CLI/stat counters.
**State/data changes:** New small bot prefs file in MeshCore storage; known bot registry loaded from persisted config.
**Edge cases:** Missing filesystem, storage full, corrupt prefs, invalid CRC, channel renamed after prefs saved, duplicate known bot key, CLI command too long, unsupported companion app ignoring bot frames.
**Acceptance criteria:**
- Existing NodePrefs load/save code remains binary-compatible.
- Bot prefs round-trip tests pass.
- CLI can enable/disable bot and list stats/known bots in a bounded response.
- Corrupt bot prefs default safely and do not crash boot.
**Verification commands:**
- `python3 tests/firmware_bot/run_tests.py`
- `grep -n "loadPrefsInt\|savePrefs\|NodePrefs" vendor/MeshCore/examples/companion_radio/DataStore.cpp vendor/MeshCore/examples/companion_radio/NodePrefs.h` followed by diff review confirming existing offsets unchanged.
- If PlatformIO available: `bash scripts/build-representative.sh --compare`
**Manual validation:** Inspect CLI help/output strings for length and field clarity.
**Risks:**
- NodePrefs binary-layout changes can corrupt existing settings. Mitigated by separate versioned BotPrefs. Trace: ITEM-pitfalls-16.
- Persisting high-churn state can wear or fill flash. Mitigated by persisting config only, not runtime caches. Trace: ITEM-pitfalls-15.
**Out of scope for this step:** Companion mobile app UI, cloud provisioning, persistent command history, large stats database.
### Step 9: Add production bot build flags and private-key import/export hardening
**Goal:** Provide a production bot firmware build mode that disables private key import/export and keeps bot-related feature flags explicit across representative targets.
**Why now:** Known-bot trust makes bot identities more sensitive, and the user chose disabling key import/export by default for production bot firmware.
**Dependencies:** Bot runtime and config exist; representative build tooling exists.
**Files:**
- `vendor/MeshCore/platformio.ini`
- `vendor/MeshCore/variants/heltec_v3/platformio.ini` if target-specific flags are needed
- `vendor/MeshCore/variants/rak4631/platformio.ini` if target-specific flags are needed
- `scripts/build-representative.sh`
- `.github/workflows/firmware-build.yml`
- `patches/meshcore/0007-add-bot-build-flags-and-key-hardening.patch`
**Existing code to inspect first:**
- `vendor/MeshCore/platformio.ini` lines defining `ENABLE_PRIVATE_KEY_IMPORT` and `ENABLE_PRIVATE_KEY_EXPORT`
- `vendor/MeshCore/examples/companion_radio/MyMesh.cpp` private key import/export command handling
- Representative variant `.ini` files for build flag inheritance
**Implementation plan:**
1. Add explicit `CMESH_BOT_ENABLED` build flags for companion bot builds in the Colorado patch series.
2. Add production bot build flag behavior that omits or overrides `ENABLE_PRIVATE_KEY_IMPORT=1` and `ENABLE_PRIVATE_KEY_EXPORT=1` for bot firmware builds.
3. Preserve a clearly named development/provisioning build option only if necessary for local testing, and keep it opt-in.
4. Ensure representative Heltec v3 and RAK4631 USB/BLE builds receive consistent bot flags.
5. Add compile-time guards around private-key import/export command paths if upstream guards are not already sufficient.
6. Update build script to print whether bot and key-hardening flags are active per environment.
7. Add tests or grep-based verification that production bot build flags do not include key import/export.
**Contracts and interfaces:**
- Production bot builds disable private key import/export by default.
- Development/provisioning behavior is explicit and not the default path used for hardware flashing.
- Existing non-bot upstream builds can remain unchanged if the patch is scoped through bot flags.
**State/data changes:** Build configuration only; no runtime data changes.
**Edge cases:** PlatformIO `build_flags` inheritance not merging as expected, variant-specific flags overriding base flags, command code compiled without guarding, CI accidentally building provisioning mode.
**Acceptance criteria:**
- Representative production bot builds show `CMESH_BOT_ENABLED` active and private key import/export inactive.
- Any provisioning build is explicit and not used by default scripts.
- Firmware source still compiles without bot flag if stock compatibility is maintained.
**Verification commands:**
- `grep -R "ENABLE_PRIVATE_KEY_IMPORT\|ENABLE_PRIVATE_KEY_EXPORT\|CMESH_BOT_ENABLED" vendor/MeshCore/platformio.ini vendor/MeshCore/variants/heltec_v3/platformio.ini vendor/MeshCore/variants/rak4631/platformio.ini vendor/MeshCore/examples/companion_radio/MyMesh.cpp`
- If PlatformIO available: `bash scripts/build-representative.sh --compare`
- CI workflow dry syntax check if available via `actionlint` or manual YAML parse.
**Manual validation:** Review final build logs to confirm production flags are printed as intended.
**Risks:**
- Private key export/import enabled in trusted bot builds can allow cloned bot identities. Mitigated by default production hardening. Trace: ITEM-pitfalls-18.
- PlatformIO flag inheritance can surprise; inspect actual compile commands/build logs. Trace: ITEM-stack-8, ITEM-pitfalls-17.
**Out of scope for this step:** Full provisioning UX, physical-button-gated key migration, mobile app support.
### Step 10: Add CI enforcement, size thresholds, and resource regression checks
**Goal:** Turn the build and size reports into hard gates for the MVP and add simple static checks for firmware safety rules.
**Why now:** After bot features and build flags exist, the project needs automated protection against resource and style regressions before flashing/deployment.
**Dependencies:** Steps 2-9.
**Files:**
- `.github/workflows/firmware-build.yml`
- `scripts/check-bot-safety.sh`
- `scripts/parse-size-report.py`
- `colorado/size-baseline/*.json` if baseline is committed
- `patches/meshcore/` updated if CI-related firmware patches are needed
**Existing code to inspect first:**
- Output from first successful representative build logs
- `scripts/build-representative.sh`
- `.github/workflows/firmware-build.yml`
**Implementation plan:**
1. Capture a clean upstream/bot baseline size JSON once representative builds work.
2. Add threshold configuration for RAK4631 BLE/USB and Heltec v3 BLE/USB deltas, initially warning above 25 KB flash / 2 KB static RAM and failing above agreed hard-review thresholds unless no baseline exists.
3. Add `scripts/check-bot-safety.sh` to grep bot source for hot-path heap APIs, Arduino `String`, dynamic containers, JSON, network/API imports, and forbidden lower-layer timing dependency.
4. Wire CI to run host tests, safety checks, representative builds, size parsing, and artifact upload.
5. Ensure CI failure messages identify the env and resource that regressed.
6. Add a local `scripts/verify.sh` wrapper that runs the same non-flashing checks developers should run before review.
7. Document how to intentionally update baselines when upstream MeshCore changes.
**Contracts and interfaces:**
- `scripts/verify.sh` is the standard pre-review command.
- CI fails on host tests/safety checks/build failures.
- CI prints size deltas even when enforcement is in warn-only mode.
**State/data changes:** Optional committed size baseline; CI artifacts.
**Edge cases:** Upstream size changes without bot changes, missing baseline, parser unable to read PlatformIO output, false-positive grep hits in comments/tests, PlatformIO cache failures.
**Acceptance criteria:**
- `scripts/verify.sh` runs host tests and safety checks locally.
- CI workflow contains the representative build matrix and uploads logs/artifacts.
- Size report clearly identifies Heltec v3 and RAK4631 resource use.
**Verification commands:**
- `bash scripts/check-bot-safety.sh`
- `bash scripts/verify.sh` if PlatformIO is not required for local subset or after installing PlatformIO
- `python3 -m py_compile scripts/parse-size-report.py`
- If PlatformIO available: `bash scripts/build-representative.sh --compare`
**Manual validation:** Review CI output format and confirm a future maintainer can see exact RAK4631 headroom.
**Risks:**
- Build-only checks can miss runtime heap/queue issues. Mitigated by static safety checks and later hardware smoke. Trace: ITEM-pitfalls-3, ITEM-pitfalls-4.
- Enforcement too strict before measured baseline can block progress. Mitigated by warn-only until baseline exists. Trace: ITEM-stack-13.
**Out of scope for this step:** Full all-companion build matrix enforcement, long-duration soak tests, field RF testing.
### Step 11: Run representative builds and flash the plugged-in Heltec v3 smoke target
**Goal:** Build verified firmware artifacts, flash the connected Heltec v3, and perform the first hardware smoke test of the firmware-only bot.
**Why now:** The user authorized the plugged-in Heltec v3 as the bot node, but hardware should only be touched after code review/builds pass.
**Dependencies:** Steps 1-10 complete; PlatformIO installed locally or CI artifacts available; Heltec v3 connected by USB; build artifacts produced.
**Files:**
- No source files unless small build/flash script fixes are discovered
- `out/firmware/*` artifacts
- `out/size/*` logs
- Optional `scripts/flash-heltec-v3.sh`
- `.forge/steps/step-11-plan.md` will contain the exact port and flash command discovered at runtime
**Existing code to inspect first:**
- `vendor/MeshCore/build.sh` flash/upload behavior
- `vendor/MeshCore/variants/heltec_v3/platformio.ini`
- Current serial devices under `/dev/cu.*` and `/dev/tty.*`
**Implementation plan:**
1. Run `scripts/verify.sh` and representative builds, requiring Heltec v3 USB/BLE and RAK4631 USB/BLE success or explicitly recorded blockers.
2. Identify the connected Heltec v3 serial port using non-destructive local device listing.
3. Choose the correct artifact/env for the user's connected Heltec v3 mode, likely `Heltec_v3_companion_radio_usb` first unless BLE-specific flashing is requested.
4. Flash the Heltec v3 with the verified production bot build, using upstream PlatformIO/build tooling rather than ad-hoc binary writes.
5. Open a serial monitor or CLI only as needed to confirm boot, firmware version/build flags, bot config status, storage stats, and no immediate crash loop.
6. Perform safe smoke commands through the available interface: DM/private if possible, `#bot`/`#testing` if available, confirm Public normal command ignore, and test emergency forwarding only if it will not surprise real users on the live mesh.
7. Record hardware smoke outcomes and any limitations in the final Forge summary.
**Contracts and interfaces:**
- Flashing uses production bot build with key export/import disabled.
- No destructive device operations beyond firmware flashing authorized by the user.
- Emergency live test is skipped or simulated if it would broadcast unexpectedly on the real mesh.
**State/data changes:** The connected Heltec v3 firmware is overwritten with the new bot firmware; local build artifacts and logs are created.
**Edge cases:** Multiple serial devices, bootloader mode required, wrong env selected, PlatformIO upload failure, device already configured for live Public channel, emergency test could broadcast to real mesh.
**Acceptance criteria:**
- Representative builds pass or blockers are documented before flashing.
- Heltec v3 flashes successfully and boots.
- Firmware reports bot enabled/configurable and storage within budget.
- Safe smoke tests show DM/allowed-channel behavior and Public normal silence.
**Verification commands:**
- `bash scripts/verify.sh`
- `bash scripts/build-representative.sh --compare`
- `ls /dev/cu.* /dev/tty.*`
- Flash command selected by implementation after reading upstream tooling, likely `pio run -d vendor/MeshCore -e Heltec_v3_companion_radio_usb -t upload --upload-port <port>` or equivalent upstream `build.sh` workflow.
- Serial/CLI command selected after detecting the connected interface.
**Manual validation:** Observe device boot logs, confirm no crash loop, confirm bot command behavior through safe channels, and avoid live emergency broadcast unless deliberately intended.
**Risks:**
- Flashing overwrites the user's current Heltec firmware/config. User has authorized flashing after pass, but the step should still report the exact action before execution. Trace: PROJECT.md.
- Live emergency testing could broadcast to Public. Mitigated by simulation or explicit pre-test confirmation if connected to a real mesh. Trace: ITEM-pitfalls-8.
**Out of scope for this step:** Field deployment across multiple bots, RAK4631 hardware flashing, all companion device flashing.
## Cross-Step Integration Checks
- Apply patch queue from a clean submodule and verify all patches apply in order.
- Run host bot tests after every behavior step.
- Run `scripts/check-bot-safety.sh` after bot source is introduced.
- Run representative builds for Heltec v3 USB/BLE and RAK4631 USB/BLE once PlatformIO is available.
- Compare bot size deltas against RAK4631 and Heltec budgets.
- Confirm no modifications were made to MeshCore routing/dispatcher/ACK/path duplicate logic.
- Confirm Public normal command traffic remains silent after all command/coordinator/config steps.
- Confirm emergency forwarding bypasses normal suppression after coordinator integration.
- Confirm production bot build disables private key import/export before flashing.
- Confirm patch export/reapply works before each commit that modifies submodule content.
## Testing Strategy
- **Host unit tests:** Parser, command classification, channel policy, emergency formatting, fingerprinting, known bot registry, coordinator FSM, BotPrefs serialization, malformed inputs, and output length caps.
- **Static safety checks:** No hot-path heap allocation or dynamic containers in bot source; no direct lower-layer `txdelay`/`rxdelay` dependency; no API/network/SQLite/plugin imports.
- **Firmware builds:** Representative local/CI builds for `Heltec_v3_companion_radio_usb`, `Heltec_v3_companion_radio_ble`, `RAK_4631_companion_radio_usb`, and `RAK_4631_companion_radio_ble`.
- **Size reports:** PlatformIO RAM/flash section output plus artifact bytes, compared against baseline and thresholds.
- **Hardware smoke:** Flash connected Heltec v3 only after build/review pass; verify boot, storage stats, bot config, allowed-channel behavior, Public silence, and safe emergency formatting if not live-broadcasting.
- **Review gates:** Every implementation step is staged and reviewed by a Claude forge-reviewer before commit; final full-project review runs after all steps.
## Out of Scope
- Full Python `meshcore-bot` feature parity, especially HTTP/API/weather/AQI/sports/satellite/Discord/web viewer/SQLite/dynamic plugin features.
- VPS coordinator emulation or dependency on internet/IP connectivity.
- Explicit on-air claim frames in v1.
- Changes to MeshCore core routing, dispatcher, ACK/path maintenance, packet duplicate tables, or lower-layer TX/RX delay semantics.
- Mobile app UI changes for bot config.
- RAK4631 hardware flashing unless separately authorized and available.
- Full all-companion build matrix before representative Heltec v3 and RAK4631 gates pass.

66
.forge/PROJECT.md Normal file
View File

@@ -0,0 +1,66 @@
# Forge Project
## Description
Build firmware for MeshCore bots using the standard MeshCore bot project (https://github.com/agessaman/meshcore-bot.git) and MeshCore companion firmware (https://github.com/meshcore-dev/MeshCore.git). The goal is to reduce bot adverts and evaluate/build a built-in response coordinator in firmware as a better alternative to the current VPS coordinator.
## Constraints
- Use the upstream MeshCore bot project and MeshCore companion firmware as primary references.
- Target all MeshCore companion device types where feasible.
- Perform only read-only discovery on local GitHub projects and the hosts `cj-vps` and `cjvana@10.0.0.222` unless later authorized otherwise.
- Deliverable type: code.
## Context
- Greenfield/Brownfield: New project in the Colorado Mesh org, starting from an empty local directory.
- Platform: MeshCore companion firmware / embedded firmware for companion device types.
- Deliverable type: code
- Date: 2026-05-14
## Initial Questions and Answers
- What kind of deliverable should Forge produce? Code project.
- New project or existing local project? New project in the Colorado Mesh org.
- Which target hardware should the first firmware build support? All companion types.
- May local projects and named hosts be inspected? Read-only discovery is OK.
## Candidate Coordination Approach
The user suggested using configured transmit and receive delays to bias which bot responds, potentially approximating which bot is closest or most appropriate for the original message. Proposed topology tiers:
- HILLTOP: highest elevation / backbone infrastructure, typical neighbors 20+, commands `set txdelay 2`, `set direct.txdelay 2`, `set rxdelay 3`.
- FOOTHILLS: mid elevation bridge nodes, typical neighbors 10-20, commands `set txdelay 1.5`, `set direct.txdelay 1`, `set rxdelay 3`.
- SUBURBAN: average rooftop installs, typical neighbors 5-10, commands `set txdelay 0.8`, `set direct.txdelay 0.4`, `set rxdelay 3`.
- LOCAL: low elevation / immediate area, typical neighbors 1-3, commands `set txdelay 0.3`, `set direct.txdelay 0.1`, `set rxdelay 3`.
- MOBILE: variable elevation, should defer to fixed infrastructure, commands `set txdelay 3`, `set direct.txdelay 2.5`, `set rxdelay 3`.
Research and planning should evaluate whether this delay-tier model can reduce duplicate bot adverts/responses, whether it conflicts with MeshCore's existing TX/RX timing semantics, and whether extra coordinator messages are still needed. The current preference is to use these tiers as inspiration only and avoid directly overloading MeshCore `txdelay`, `direct.txdelay`, or `rxdelay` for bot election.
## Refined Decisions
- Scope pivot: target a firmware-only bot rather than a host-side meshcore-bot permission layer. Use meshcore-bot as behavioral reference, not the runtime dependency, unless storage/feasibility research proves firmware-only is impractical.
- Coordination scope: all bot-originated traffic is in scope, but bot traffic should not use the general Public channel during normal operation.
- Channel policy: firmware bot may handle private DMs, #bot, and #testing. Messages sent to #emergency should be forwarded/announced to Public as `EMERGENCY MESSAGE FROM <user>` followed by the original message text.
- Delay policy: use separate bot-specific coordinator delays and suppression windows; do not repurpose lower-layer MeshCore timing knobs as the primary coordination mechanism.
- Duplicate/latency target: balanced; accept rare duplicates while reducing bot noise without excessive response latency.
- Claims: defer explicit on-air claim frames in the first implementation; use passive listen-before-answer suppression first.
- Trust: suppression should trust known bot identities only.
- Compatibility: new firmware-only bot nodes may require the new firmware; mixed stock compatibility is not a hard requirement for participating bots.
- Off-grid model: design for decentralized firmware operation, not VPS parity.
- Source strategy: keep upstream MeshCore as a submodule and maintain Colorado Mesh firmware-bot code/patches around it.
- Build validation: use a small representative build set during development, specifically Heltec v3 and RAK4631, then expand later.
- Additional research request: estimate flash/RAM/storage required by a firmware-only bot and compare it with space available on representative companion devices.
- Hardware available for validation: a Heltec v3 is plugged into this machine and is intended to be the user's bot node. The user authorized using it for firmware flashing after implementation and verification pass.
## Final Deep-Questioning Decisions
- Firmware v1 should pursue more bot parity than the compact MVP, focused on lightweight fun + utility commands rather than HTTP/API/database-backed features.
- RAK4631 remains a hard release gate despite Heltec v3 being the first physical bot node.
- Bot configuration should be runtime-editable through CLI/config commands in Phase 1.
- Production bot firmware should disable private key import/export by default.
- Emergency forwarding may use multipart Public messages rather than strict one- or two-packet truncation.
- Emergency forwarding should never be suppressed; duplicate emergency Public announcements are preferable to a missed emergency.
- Suppression metadata should be hidden if MeshCore supports it; otherwise use passive recognition rather than visible chat markers.
- CI may install PlatformIO and must build representative Heltec v3 USB/BLE plus RAK4631 USB/BLE targets with size reports.
- Public channel bot commands should be ignored silently except for #emergency routing to Public.
- The plugged-in Heltec v3 should be flashed after code review and builds pass.
## Open Planning Interpretations
- “More bot parity” means prioritize firmware-feasible fun + utility commands and avoid features requiring network APIs, files, databases, plugins, or large dynamic text unless size evidence proves they fit.
- “Runtime CLI” should be implemented as compact bot-specific CLI/config commands without changing existing `NodePrefs` binary layout.
- “Multipart emergency” must still be bounded by rate limits and loop prevention to avoid accidental Public floods.
- “All companion types” means representative Heltec v3 and RAK4631 gates first, then expansion to the full companion build matrix after MVP viability.

261
.forge/research/SYNTHESIS.md Normal file
View File

@@ -0,0 +1,261 @@
# Research Synthesis
## Status
- Files synthesized: stack.md, pitfalls.md, architecture.md, prior-art.md, codex-analysis.md, PROJECT.md
- Files missing: none
- Overall confidence: HIGH
## Executive Summary
This project is now a firmware-only MeshCore companion bot, not a host-side bot permission layer and not a VPS coordinator replacement by emulation. The proven path is to keep upstream `meshcore-dev/MeshCore` as the firmware/build substrate, use Arduino C++/PlatformIO exactly as upstream does, and add a compact Colorado Mesh bot runtime at the companion application layer. `agessaman/meshcore-bot`, the Colorado VPS/community coordinator, and stale Codex coordinator-first notes are useful only as behavioral/prior-art references; they must not drive the runtime architecture.
The recommended implementation is a small embedded command responder plus passive response coordinator inside `examples/companion_radio/MyMesh`: receive DMs and allowed channel messages, classify commands, enforce channel policy, compute semantic fingerprints, schedule bounded delayed responses, cancel when a trusted known bot has already answered, and send via existing MeshCore `sendMessage()` / `sendGroupMessage()` APIs. Normal bot traffic should be limited to private DMs, `#bot`, and `#testing`; `#emergency` should use a dedicated Public forwarding path with the exact required emergency announcement semantics and strict dedup/rate limiting. Do not repurpose MeshCore `txdelay`, `direct.txdelay`, or `rxdelay` for bot election; use bot-specific role/delay/suppression settings.
The top risks are RAK4631 flash/RAM limits, heap fragmentation, packet-pool exhaustion, unsafe text parsing, wrong channel routing, spoofed suppression, emergency amplification, hidden nodes, and companion protocol breakage. Mitigate by designing to RAK4631 first, keeping v1 commands deterministic and single-packet, using fixed static state with no hot-path heap allocation, keeping runtime suppression state in RAM, persisting only a small versioned `BotPrefs`, and measuring PlatformIO size deltas in CI. The user's plugged-in Heltec v3 is the correct first hardware smoke target because it is available and has generous flash, but RAK4631 BLE remains the release gate.
## Key Decisions (resolved by research)
1. **Build in MeshCore's firmware stack.** Use upstream MeshCore Arduino C++/PlatformIO and current upstream dependency pins; do not introduce ESP-IDF-only, Zephyr, Rust, embedded Python/JS, MQTT firmware logic, SQLite, or a custom radio stack. Source refs: ITEM-stack-1, ITEM-stack-4, ITEM-prior-art-1.
2. **Use a submodule plus patch/overlay workflow.** Keep `meshcore-dev/MeshCore` pinned as an upstream submodule and maintain Colorado bot code as a small patch/overlay series applied during builds. Source refs: ITEM-stack-2, ITEM-architecture-14.
3. **Implement a firmware-resident bot, not host-side coordination.** Phase 1 should generate selected bot responses directly in firmware; `meshcore-bot` is a behavior oracle only. Source refs: ITEM-stack-6, ITEM-prior-art-3, ITEM-pitfalls-2, ITEM-architecture-8.
4. **Place bot logic at the companion application layer.** Integrate with `MyMesh` callbacks and loop; leave `Dispatcher`, `Mesh::routeRecvPacket()`, ACKs, path returns, retransmit timing, and packet duplicate tables untouched. Source refs: ITEM-architecture-1, ITEM-stack-5.
5. **Use strict channel policy.** Normal bot handling is DMs, `#bot`, and `#testing`; Public is ignored except for the dedicated `#emergency` forwarding path. Source refs: ITEM-architecture-3, ITEM-stack-7, ITEM-prior-art-12, ITEM-pitfalls-7.
6. **Forward emergencies explicitly and safely.** `#emergency` should produce Public text beginning `EMERGENCY MESSAGE FROM <user>` and preserve/truncate original text with bounded one- or two-packet formatting, deduplication, loop prevention, and rate limiting. Source refs: ITEM-architecture-4, ITEM-prior-art-12, ITEM-pitfalls-8.
7. **Coordinate with passive listen-before-answer first.** Use semantic fingerprints, bounded delayed sends, and cancellation on trusted known-bot responses. Do not add claim frames in v1. Source refs: ITEM-architecture-5, ITEM-architecture-6, ITEM-pitfalls-9, ITEM-pitfalls-10.
8. **Trust known bot identities only.** Suppression must be keyed to known bot public keys where possible; plain group text names are spoofable hints and must not suppress emergency forwarding. Source refs: ITEM-architecture-7, ITEM-pitfalls-11.
9. **Design storage and RAM for RAK4631.** Heltec v3 is comfortable, but the nRF52840 RAK4631 app/RAM limits define the portable v1 budget. Source refs: ITEM-stack-9, ITEM-stack-10, ITEM-stack-11, ITEM-stack-12, ITEM-architecture-10, ITEM-pitfalls-1.
10. **Instrument and size-gate from the start.** CI should build Heltec v3 USB/BLE and RAK4631 USB/BLE, capture PlatformIO size output, and enforce bot delta thresholds after baseline. Source refs: ITEM-stack-8, ITEM-stack-13, ITEM-architecture-13, ITEM-pitfalls-1.
11. **Start hardware smoke tests on the plugged-in Heltec v3.** Use the local Heltec v3 for first flash/build smoke only after the implementation plan is approved; expand to RAK4631 before release. Source refs: PROJECT.md, ITEM-stack-8, ITEM-stack-9.
## Questions for User
### Q-1: Which exact firmware MVP commands should ship first?
- **Category:** scope
- **Why it matters:** Every command adds flash, response strings, parser cases, tests, and possible airtime. The v1 command set determines whether the feature remains firmware-feasible on RAK4631.
- **Default recommendation:** Ship `ping`, `test`, `help/cmd`, `about`, `status` with battery/storage, simple `dice/roll`, DM replies, `#bot`/`#testing` replies, and `#emergency` forwarding. Defer weather, AQI, sports, jokes via web APIs, satellite/solar forecasts, Discord/web viewer, SQLite stats, and repeater-management workflows.
- **Source refs:** ITEM-stack-6, ITEM-prior-art-3, ITEM-architecture-8, ITEM-pitfalls-2
- **Priority:** HIGH
### Q-2: Should the bot be enabled by default after flashing, or require explicit enable/configuration?
- **Category:** ux
- **Why it matters:** A firmware bot that responds immediately after flashing can surprise users or create channel noise if channels/known bots/tier are not configured correctly.
- **Default recommendation:** Default to enabled for DMs and local smoke-test commands only, but require explicit channel policy resolution and role configuration before responding on `#bot`/`#testing`; always require explicit `#emergency` source/destination channel validation before Public forwarding.
- **Source refs:** ITEM-architecture-3, ITEM-architecture-9, ITEM-pitfalls-7, ITEM-pitfalls-16
- **Priority:** HIGH
### Q-3: What should the default response role/tier be for the user's plugged-in Heltec v3 bot node?
- **Category:** ux
- **Why it matters:** The first hardware smoke target is also intended to become the user's bot node. Its default role affects response latency, suppression behavior, and whether it defers to fixed infrastructure.
- **Default recommendation:** Configure the plugged-in Heltec v3 as `LOCAL` or `SUBURBAN` for smoke tests, with a conservative 300-1500 ms response window; change role later based on deployment location.
- **Source refs:** PROJECT.md, ITEM-stack-9, ITEM-architecture-12, ITEM-pitfalls-14
- **Priority:** HIGH
### Q-4: What duplicate-response and latency targets are acceptable for v1?
- **Category:** constraints
- **Why it matters:** Passive suppression can reduce noise but cannot guarantee exactly-once behavior in hidden-node topologies. The target determines delay windows and whether claim frames become necessary later.
- **Default recommendation:** Accept rare duplicates in v1. Target p95 added latency under 3 seconds for normal channel commands and duplicate bot replies under 5-10% in field scenarios, with metrics before adding claim frames.
- **Source refs:** ITEM-architecture-6, ITEM-pitfalls-9, codex-analysis
- **Priority:** HIGH
### Q-5: How should known bot identities be provisioned and maintained?
- **Category:** technical
- **Why it matters:** Suppression should trust known bot identities only, but firmware needs a bounded, field-serviceable way to store and update public keys.
- **Default recommendation:** Store 16 known bot full public keys plus optional labels/capability flags in versioned `BotPrefs`; expose rescue CLI commands and later companion protocol commands to add/remove/list keys.
- **Source refs:** ITEM-architecture-7, ITEM-architecture-9, ITEM-architecture-10, ITEM-pitfalls-11
- **Priority:** HIGH
### Q-6: Should unauthenticated group-text responses ever suppress a local pending response?
- **Category:** risk
- **Why it matters:** MeshCore group text names are spoofable. Suppressing based on text alone can silence the real bot or block emergency forwarding.
- **Default recommendation:** For v1, allow weak group-text suppression only for low-risk normal `#bot`/`#testing` replies that match known bot labels and fingerprints; never use weak hints for `#emergency` forwarding or safety-critical behavior.
- **Source refs:** ITEM-architecture-7, ITEM-pitfalls-11, ITEM-pitfalls-8
- **Priority:** HIGH
### Q-7: What exact `#emergency` behavior should occur when the original text is too long?
- **Category:** ux
- **Why it matters:** MeshCore text/frame limits mean the required emergency header plus original text may not fit in one message. Truncation policy affects safety and airtime.
- **Default recommendation:** Use a two-stage formatter when needed: first Public packet `EMERGENCY MESSAGE FROM <user>`, second Public packet containing the original text truncated to fit with an explicit truncation marker. Do not send unlimited multipart emergency messages in v1.
- **Source refs:** ITEM-architecture-4, ITEM-pitfalls-5, ITEM-pitfalls-8, ITEM-prior-art-2
- **Priority:** HIGH
### Q-8: What channel names and matching rules should be canonical?
- **Category:** technical
- **Why it matters:** Channel indexes are local and can change; substring matching could make the bot respond on unintended channels such as `#botnet`.
- **Default recommendation:** Use exact normalized channel names `Public`, `#bot`, `#testing`, and `#emergency`; resolve to channel details at boot and after channel changes; refuse channel operation if names are missing, duplicated, or ambiguous.
- **Source refs:** ITEM-architecture-3, ITEM-pitfalls-7
- **Priority:** HIGH
### Q-9: What persistent configuration surface is acceptable for v1?
- **Category:** technical
- **Why it matters:** Appending to `NodePrefs` can break existing binary layout, but compile-time-only config is hard to field-tune.
- **Default recommendation:** Add a separate versioned `/bot_prefs_v1` file with magic/version/length/CRC for enable flag, tier, delay/jitter, cooldowns, allowed channels, emergency mapping, known bots, command bitset, and small stats reset marker. Do not insert fields into existing `NodePrefs` offsets.
- **Source refs:** ITEM-architecture-9, ITEM-pitfalls-15, ITEM-pitfalls-16, ITEM-prior-art-10
- **Priority:** HIGH
### Q-10: What are the hard size budgets for accepting v1?
- **Category:** constraints
- **Why it matters:** Research estimates are feasible but unmeasured locally because PlatformIO was unavailable. The plan needs explicit thresholds to prevent feature creep.
- **Default recommendation:** Treat RAK4631 BLE as the gate: aim for <=25 KB incremental app flash and <=2 KB static RAM initially; hard-review anything above 40-60 KB flash or 4 KB RAM. Persistent bot config should stay under 4 KB. Capture exact deltas in CI.
- **Source refs:** ITEM-stack-10, ITEM-stack-11, ITEM-stack-12, ITEM-stack-13, ITEM-architecture-10, ITEM-architecture-11, ITEM-pitfalls-1
- **Priority:** HIGH
### Q-11: Should private key import/export be disabled in production bot builds?
- **Category:** risk
- **Why it matters:** Known-bot trust makes bot identities valuable. If private keys remain exportable/importable, a compromised host or BLE session can clone a trusted bot identity.
- **Default recommendation:** Disable private key export/import for production bot firmware. Use a separate provisioning build or a physical-button-gated temporary window if identity migration is required.
- **Source refs:** ITEM-pitfalls-18
- **Priority:** HIGH
### Q-12: What host/app protocol extensions are needed in v1, if any?
- **Category:** technical
- **Why it matters:** New companion protocol frames can break apps if they collide with existing commands or appear unexpectedly. But field config and stats need some access path.
- **Default recommendation:** Keep v1 usable with rescue CLI plus minimal queried stats/config frames only. Gate all new protocol commands by firmware capability/version and never change existing frame semantics.
- **Source refs:** ITEM-architecture-9, ITEM-architecture-13, ITEM-pitfalls-13
- **Priority:** MEDIUM
### Q-13: Which hardware targets define “done” for the first release candidate?
- **Category:** constraints
- **Why it matters:** The project wants all companion device types eventually, but first release quality needs a focused representative set.
- **Default recommendation:** Inner-loop builds: Heltec v3 USB, Heltec v3 BLE, RAK4631 USB, RAK4631 BLE. First hardware smoke: the plugged-in Heltec v3. Release gate: RAK4631 BLE builds and passes size/stress tests. Expand all companion builds after representative success.
- **Source refs:** PROJECT.md, ITEM-stack-8, ITEM-stack-9, ITEM-stack-10, ITEM-pitfalls-17
- **Priority:** MEDIUM
### Q-14: How much telemetry should be exposed to users versus debug tooling?
- **Category:** ux
- **Why it matters:** Operators need visibility into suppression and hidden nodes, but normal users should not see noisy internal state over LoRa.
- **Default recommendation:** Expose compact local stats through CLI/protocol (`observed`, `sent`, `suppressed`, `duplicates`, `emergency forwards`, queue failures, storage used/total). Do not post coordinator diagnostics into normal channels except explicit debug commands.
- **Source refs:** ITEM-architecture-13, ITEM-pitfalls-12, ITEM-pitfalls-4
- **Priority:** MEDIUM
### Q-15: Should future explicit claim frames be planned in the protocol now, even if disabled in v1?
- **Category:** technical
- **Why it matters:** Planning the fingerprint/state model now can avoid repainting the architecture later, but implementing claims immediately adds airtime and spoofing risks.
- **Default recommendation:** Reserve state-machine hooks and stats for `known_claim_seen`, but do not transmit claim frames in v1. If later added, claims must be compact, versioned, length-checked, fingerprint-bound, scoped, and authenticated by known bot identity.
- **Source refs:** ITEM-architecture-6, ITEM-architecture-7, ITEM-pitfalls-9, ITEM-pitfalls-11
- **Priority:** MEDIUM
## Technical Direction
### Stack
- **Firmware/runtime:** C++ in upstream MeshCore companion firmware, Arduino framework, PlatformIO, upstream dependency pins preserved. Source refs: ITEM-stack-1, ITEM-stack-4.
- **Repository shape:** wrapper repo with pinned `upstream/MeshCore` submodule, Colorado overlay files, and deterministic patch queue applied into a build worktree. Source refs: ITEM-stack-2, ITEM-architecture-14.
- **Target baseline:** current MeshCore main pinned intentionally, not stale local-only assumptions. Source refs: ITEM-stack-3.
- **Build matrix:** use upstream `build.sh`; inner loop builds `Heltec_v3_companion_radio_usb`, `Heltec_v3_companion_radio_ble`, `RAK_4631_companion_radio_usb`, and `RAK_4631_companion_radio_ble`; later run all companion builds. Source refs: ITEM-stack-8.
- **Firmware coding style:** fixed-size arrays/ring buffers, compile-time flags, no heap allocation in receive/parse/schedule/send paths, bounded `char[]` parsing, terse single-packet responses. Source refs: ITEM-stack-5, ITEM-pitfalls-3, ITEM-pitfalls-5, ITEM-pitfalls-6.
- **Host-side tooling:** Python only for tests, fixtures, worktree/patch tooling, simulations, and size-report parsing. No Python runtime dependency in deployed firmware. Source refs: ITEM-stack-14.
### Architecture
The bot should be a `FirmwareBot`/`BotRuntime` owned by companion `MyMesh`. `MyMesh` adapts MeshCore callbacks into compact `BotMessage` inputs, invokes policy/classification/execution/coordination, and sends approved `BotResponse` outputs through existing MeshCore send APIs. Core modules should be:
1. `BotPolicy`: enabled state, exact channel policy, known bots, command bitset, emergency rules, rate limits.
2. `CommandClassifier`: bounded normalization, prefix/verb matching, semantic fingerprint inputs.
3. `CommandExecutor`: fixed C++ handlers for v1 commands with short response buffers.
4. `ResponseCoordinator`: passive listen-before-answer FSM and fixed pending/recent tables.
5. `EmergencyForwarder`: dedicated `#emergency` to Public path with dedup, loop prevention, and bounded formatting.
6. `BotPrefs` / `KnownBotRegistry` / `BotStats`: small persisted config, explicit trusted keys, and metrics.
Recommended v1 data flow:
1. MeshCore receives/decrypts a DM or group message and invokes `MyMesh` callbacks.
2. `BotPolicy` rejects all normal Public traffic and only allows DMs, `#bot`, and `#testing`; `#emergency` is diverted to `EmergencyForwarder`.
3. `CommandClassifier` parses `(const char*, length)`, normalizes only safe ASCII command verbs, and derives a 64-bit semantic fingerprint from stable application fields.
4. `CommandExecutor` creates a short candidate response without allocating a MeshCore packet.
5. `ResponseCoordinator` schedules the response in a fixed table using role/delay/jitter/score. It does not allocate a packet until the timer wins.
6. If a trusted known bot response for the same fingerprint is heard first, the pending response is suppressed and stats are updated.
7. When due, `MyMesh` sends a single concise DM/group message via existing APIs.
8. Runtime fingerprints, suppression windows, emergency recent cache, and counters remain bounded and mostly volatile.
### Storage / Flash / RAM Estimates
| Target | Hardware/app region facts | Current signal from research | Recommended v1 bot budget | Implication |
|---|---|---:|---:|---|
| Heltec WiFi LoRa 32 V3 | ESP32-S3, 8 MB flash, 512 KB SRAM, no PSRAM; PlatformIO default 8 MB partition gives ~3,342,336 bytes per OTA app slot and ~1,572,864 bytes SPIFFS | v1.15 release app signals around 615 KB USB / ~1.2 MB BLE non-merged; Heltec has wide flash margin | 10-25 KB flash target, hard review above 40-60 KB; 0.5-4 KB RAM; 0-4 KB persistent bot prefs | Comfortable first smoke target. Do not optimize for Heltec alone because it hides RAK constraints. Source refs: ITEM-stack-9, ITEM-stack-11, ITEM-stack-12, ITEM-pitfalls-1 |
| RAK4631 | nRF52840, 1 MB flash, 256 KB RAM; MeshCore companion app capped at 712,704 bytes and RAM region ~237,568 bytes after SoftDevice/linker reservations | v1.15 UF2 assets ~933-949 KB container / ~467-475 KB zip proxy; exact ELF headroom requires local PlatformIO build | Aim <=25 KB additional flash and <=2 KB static RAM initially; design under 4 KB runtime RAM and under 4 KB filesystem storage; hard review above 40-60 KB flash or 4-8 KB RAM | Release gate and limiting target. No large help text, APIs, databases, logs, dynamic plugins, or persistent histories. Source refs: ITEM-stack-10, ITEM-stack-11, ITEM-stack-12, ITEM-architecture-10, ITEM-architecture-11, ITEM-pitfalls-1 |
Concrete compact state target: 8-12 pending coordinator entries, 16-24 recent fingerprint entries, 16 known bot keys, counters under 256 B, and one shared 160-byte response scratch buffer. Persistent storage should contain only versioned config and known bot keys; suppression history and emergency caches should be RAM-only.
### Prior Art to Leverage
- **Upstream MeshCore companion firmware:** patch base, transport, storage, send APIs, callbacks, and multi-target build system. Source refs: ITEM-prior-art-1.
- **MeshCore companion protocol:** short-message and storage-report constraints; useful for future stats/config access, not a runtime bot API. Source refs: ITEM-prior-art-2.
- **agessaman meshcore-bot:** command names, channel policy concepts, cooldown/rate-limit behavior, and response wording. Do not port HTTP/API/SQLite/plugins. Source refs: ITEM-prior-art-3.
- **Colorado Mesh community/VPS bot:** observable coordination semantics: DMs bypass, channel responses coordinated, local scoring, fallback delays. Preserve concepts without preserving host infrastructure. Source refs: ITEM-prior-art-4.
- **Meshtastic firmware modules:** proof that small fixed embedded modules are the right pattern; use bounded config, static messages, rate limits, and compile-time flags. Source refs: ITEM-prior-art-5.
- **disaster.radio console:** practical model for bounded firmware command parsing. Source refs: ITEM-prior-art-6.
- **Cyclenerd meshcore-bot:** conservative scope lesson: DM/private/opt-in channels are safer than Public. Source refs: ITEM-prior-art-8.
- **LoRa APRS and MESH-API/MESH-AI:** conceptual lessons for duplicate buffers, origin markers, and loop prevention, but do not reuse GPL/heavy host code in firmware. Source refs: ITEM-prior-art-7, ITEM-prior-art-9.
## Detailed Planning Implications
1. **Repository initialization:** create wrapper repo structure, add/pin MeshCore submodule, define overlay/patch application script, and document exact upstream SHA. Verify upstream builds before bot patches. Source refs: ITEM-stack-2, ITEM-stack-3, ITEM-architecture-14.
2. **Build/tooling gate:** add `scripts/build-representative.sh` to apply patches, build Heltec v3 USB/BLE and RAK4631 USB/BLE, parse PlatformIO size output, and archive artifacts. Source refs: ITEM-stack-8, ITEM-stack-13, ITEM-pitfalls-1.
3. **First smoke target:** after plan approval, use the plugged-in Heltec v3 for first firmware smoke because it is locally available and has generous flash. Keep flashing steps explicit and reversible. Source refs: PROJECT.md, ITEM-stack-9.
4. **RAK4631 release gate:** do not declare firmware feasibility complete until RAK4631 BLE builds with measured app/RAM headroom and passes stress tests. Source refs: ITEM-stack-10, ITEM-pitfalls-1.
5. **Patch boundaries:** keep patches small: bot source files, `MyMesh` hook/adapters, `BotPrefs`/DataStore integration, optional CLI/protocol constants, and CI/build flags. Source refs: ITEM-architecture-14.
6. **Parser first:** implement bounded `(ptr,len)` parser/normalizer with unit/fuzz cases for max-length messages, missing colon, repeated colon, UTF-8, NUL, controls, empty commands, and exact channel names. Source refs: ITEM-pitfalls-6.
7. **Policy before execution:** implement exact channel resolution and disabled/conservative defaults before adding command handlers so Public noise is structurally impossible. Source refs: ITEM-architecture-3, ITEM-pitfalls-7.
8. **Emergency path as its own step:** implement `EmergencyForwarder` separately from normal command execution, with dedup TTL, loop prevention, formatting tests, and rate-limit counters. Source refs: ITEM-architecture-4, ITEM-pitfalls-8.
9. **Command MVP:** add short deterministic command handlers and golden behavior fixtures from `meshcore-bot`; ensure every output fits MeshCore text constraints with node-name prefix. Source refs: ITEM-stack-6, ITEM-pitfalls-5, ITEM-prior-art-3.
10. **Coordinator FSM:** implement fixed-size passive suppression after basic command handling works; defer any claim frames. Verify all state transitions, TTL expiry, queue-full, and cancel-on-known-response paths. Source refs: ITEM-architecture-6, ITEM-pitfalls-4, ITEM-pitfalls-9.
11. **Known bot registry:** implement full-key storage, labels, bounded capacity, and trust decisions before allowing suppression across nodes. Source refs: ITEM-architecture-7, ITEM-pitfalls-11.
12. **Persistence isolation:** add versioned `BotPrefs`; do not mutate existing `NodePrefs` binary layout. Default disabled/safe on missing, corrupt, or incompatible prefs. Source refs: ITEM-architecture-9, ITEM-pitfalls-16.
13. **Instrumentation:** add `BotStats` from day one, including observed, eligible, scheduled, sent, suppressed, weak-suppressed, emergency forwarded, duplicate emergency, queue/pool failures, and storage totals. Source refs: ITEM-architecture-13.
14. **No hot-path heap:** add review/static checks forbidding `String`, `malloc/new`, `std::vector`, maps, JSON builders, and variable heap strings in bot receive/parse/schedule/send code. Source refs: ITEM-pitfalls-3.
15. **Protocol compatibility:** keep new companion protocol frames optional/query-only in v1; no unsolicited new frames to old apps unless version/capability-gated. Source refs: ITEM-pitfalls-13.
16. **Security/release flags:** decide production key export/import policy before distributing trusted bot firmware. Source refs: ITEM-pitfalls-18.
## Risk Register
| Priority | Risk | Impact | Mitigation | Source refs |
|---|---|---|---|---|
| CRITICAL | RAK4631 BLE exceeds flash/RAM | Build failure or unstable constrained target | Treat RAK4631 BLE as gate; CI size deltas; cap v1 at compact command set | ITEM-pitfalls-1, ITEM-stack-10, ITEM-stack-12 |
| CRITICAL | Python bot feature creep | Firmware bloat, off-grid failure, unmaintainable partial clone | Firmware MVP only; Python as behavior oracle; defer API/web/SQLite features | ITEM-pitfalls-2, ITEM-stack-6, ITEM-prior-art-3 |
| CRITICAL | Heap fragmentation | Long-running crashes, packet allocation failures, BLE instability | Fixed arrays/rings, no hot-path heap, bounded buffers, no `String`/JSON | ITEM-pitfalls-3 |
| CRITICAL | Packet pool/offline queue exhaustion | Dropped user traffic and failed sends | Keep pending responses separate from packets; allocate only when timer wins; small fixed pending table | ITEM-pitfalls-4 |
| CRITICAL | Unsafe parsing | Buffer errors, command misfires, corrupted output | `(ptr,len)` parser, explicit NUL bounds, `snprintf`, fuzz/unit tests | ITEM-pitfalls-6 |
| CRITICAL | Wrong channel routing | Bot answers on Public or misses emergency policy | Exact channel name/hash resolution, diagnostics, refuse ambiguous config | ITEM-pitfalls-7, ITEM-architecture-3 |
| CRITICAL | Emergency forwarding loops/amplification | Public flood during emergencies | Separate emergency path, idempotency TTL, no recursive forwarding, rate limits | ITEM-pitfalls-8, ITEM-architecture-4 |
| CRITICAL | Spoofed suppression | Malicious/mistaken node silences real bots | Known bot public keys; weak hints only for low-risk channels; no weak emergency suppression | ITEM-pitfalls-11, ITEM-architecture-7 |
| CRITICAL | Blocking companion loop | BLE/serial/radio/display stalls | Timestamp FSM in `loop()`, no `delay()`, no busy waits | ITEM-pitfalls-12, ITEM-architecture-6 |
| CRITICAL | NodePrefs layout breakage | Existing settings corrupted | Separate versioned `BotPrefs`; default safe on invalid prefs | ITEM-pitfalls-16, ITEM-architecture-9 |
| CRITICAL | Target-specific APIs break portability | Works on Heltec only; fails nRF/RP2040/STM32 | Platform-neutral bot core behind `MyMesh`/DataStore abstractions; representative build matrix | ITEM-pitfalls-17 |
| CRITICAL | Production key export/import remains enabled | Trusted bot identity can be cloned | Disable export/import in production bot firmware or gate provisioning | ITEM-pitfalls-18 |
| MODERATE | Text/frame limits truncate replies | Lost context and high airtime from multipart chatter | Single-packet command responses; emergency two-message max when needed | ITEM-pitfalls-5, ITEM-prior-art-2 |
| MODERATE | Passive suppression cannot solve hidden nodes | Duplicate replies still occur | Set expectations, measure, deterministic jitter, optional authenticated claims later | ITEM-pitfalls-9 |
| MODERATE | Bad semantic fingerprint | False suppression or missed duplicates | Fingerprint stable app fields, not raw packet hash or text alone; tests | ITEM-pitfalls-10, ITEM-architecture-5 |
| MODERATE | RSSI/SNR overused as “closest” signal | Wrong bot wins | Use SNR/RSSI only as weak zero-hop/direct hint; tier/health/path/tiebreaker dominate | ITEM-pitfalls-14 |
| MODERATE | Flash wear/storage fill | DataStore corruption or lost contacts/channels | Persist only config; runtime caches in RAM; lazy coarse counters only | ITEM-pitfalls-15, ITEM-prior-art-10 |
## Conflicts & Tradeoffs
1. **Heltec-first smoke vs RAK-first design.** The plugged-in Heltec v3 is the right first hardware smoke target, but its 8 MB flash can hide the real release constraint. Resolution: smoke on Heltec, budget and release-gate on RAK4631 BLE. Source refs: PROJECT.md, ITEM-stack-9, ITEM-stack-10, ITEM-pitfalls-1.
2. **Full firmware bot vs Python feature parity.** Firmware-only is the project goal, but Python bot parity is not feasible on RAK4631-class MCUs. Resolution: firmware-native MVP with deterministic local commands; host/Python remains only a behavior reference and tooling source. Source refs: ITEM-stack-6, ITEM-pitfalls-2, ITEM-prior-art-3.
3. **Passive suppression vs exactly-once responses.** Passive suppression saves airtime and avoids new frames but cannot guarantee exactly-once under hidden nodes. Resolution: passive first with metrics; future authenticated claims only if field data demands. Source refs: ITEM-architecture-6, ITEM-pitfalls-9, codex-analysis.
4. **Known identity security vs group-channel realities.** Strong suppression wants full public-key identity, but plain group text carries spoofable sender names. Resolution: full-key registry for authoritative trust; weak hints only for low-risk channels; no weak emergency suppression. Source refs: ITEM-architecture-7, ITEM-pitfalls-11.
5. **Config flexibility vs storage/migration safety.** Field operators need tuning, but `NodePrefs` binary layout is fragile and storage is finite. Resolution: separate compact versioned `BotPrefs`, CLI rescue, optional protocol config later. Source refs: ITEM-architecture-9, ITEM-pitfalls-16.
6. **Emergency reliability vs airtime conservation.** Public emergency forwarding may duplicate under hidden nodes, but suppressing too aggressively can drop critical alerts. Resolution: short idempotent forwards, longer emergency dedup TTL, no recursive forwarding, tolerate identical duplicates over missed emergencies. Source refs: ITEM-architecture-4, ITEM-pitfalls-8.
7. **Topology tier policy vs actual RF quality.** LOCAL/SUBURBAN/HILLTOP/MOBILE is useful operator intent but not true proximity. Resolution: role is a delay/score bias, not the sole winner; combine with directness, path freshness, queue health, and deterministic tiebreaking. Source refs: ITEM-architecture-12, ITEM-pitfalls-14.
8. **Codex supplemental analysis vs refreshed research.** Codex recommended useful general cautions, but it was coordinator-first/stale relative to the firmware-only pivot. Resolution: use Codex only where it aligns with refreshed Claude research on passive soft coordination, hidden nodes, airtime, and embedded constraints; ignore host/coordinator-first conclusions. Source refs: codex-analysis, ITEM-stack-6, ITEM-architecture-1.
## Confidence Assessment
| Dimension | Status | Confidence | Notes |
|-----------|--------|------------|-------|
| stack | complete | HIGH | Clear firmware stack: upstream MeshCore Arduino C++/PlatformIO, submodule/patch workflow, representative builds, storage budget estimates. |
| pitfalls | complete | HIGH | Strong local-code-backed risks for RAK4631 limits, heap, queues, parsing, channel routing, emergency loops, protocol compatibility, and keys. |
| architecture | complete | HIGH | Clear companion-layer module architecture, policy/classifier/executor/coordinator split, BotPrefs, known bot registry, stats, passive FSM. |
| prior-art | complete | HIGH | Strong prior art from upstream MeshCore, Python/community bots, Meshtastic modules, disaster.radio, and host-bot examples. |
| codex-analysis | complete | MEDIUM | Optional supplemental research only; stale coordinator-first framing must not override refreshed firmware-only research. Useful only for general embedded cautions. |

150
.forge/research/architecture.md Normal file
View File

@@ -0,0 +1,150 @@
# Architecture Research: Firmware-only MeshCore Bot
Project date: 2026-05-14
Mode: architecture
Scope: firmware-only MeshCore bot integrated into MeshCore companion firmware. This replaces stale coordinator-first/host-bot conclusions with an embedded command-handling architecture using upstream MeshCore as submodule/patch base and `meshcore-bot` as behavioral reference only.
### ITEM-architecture-1: Put the firmware bot at the companion application layer
- **Recommendation:** Implement the bot as a `FirmwareBot`/`BotRuntime` owned by `examples/companion_radio/MyMesh`, invoked from `onMessageRecv()`, `onSignedMessageRecv()`, `onChannelMessageRecv()`, `onControlDataRecv()`, outbound send completion/timeout paths, and `MyMesh::loop()`. Keep `mesh::Dispatcher`, `mesh::Mesh::routeRecvPacket()`, ACKs, packet duplicate tables, path returns, and retransmit timing untouched.
- **Rationale:** MeshCore already separates radio scheduling (`Dispatcher`), packet parsing/dedup/routing (`Mesh`), chat abstractions (`BaseChatMesh`), and companion UX/protocol (`MyMesh`). Bot commands are application behavior, not mesh forwarding behavior. Placing the bot in `MyMesh` gives it decrypted DM/channel text, channel index/name, ContactInfo for DMs, SNR/path metadata, storage access, and send APIs without changing core behavior for repeaters, room servers, sensors, or stock companion clients.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/src/Mesh.cpp`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not implement bot election inside `Dispatcher` or `Mesh::routeRecvPacket()`: that would couple command policy to every packet role. Do not keep a Python/VPS runtime in the critical path: the project pivot is firmware-only and off-grid.
### ITEM-architecture-2: Split the bot into policy, classification, execution, and coordination components
- **Recommendation:** Structure the embedded bot as four small modules: `BotPolicy` (DM/channel allowlist, banned users, known bots, emergency rules), `CommandClassifier` (normalization and command/keyword match), `CommandExecutor` (fixed C++ command handlers and response formatting), and `ResponseCoordinator` (delayed send/suppression state). `MyMesh` should be only the adapter that turns MeshCore callbacks into `BotMessage` inputs and sends approved `BotResponse` outputs.
- **Rationale:** The standard Python bot combines config, channel filtering, command lookup, external APIs, response formatting, and sending in host code. Firmware needs the same observable decisions but cannot afford a plugin framework or external data dependencies. Separate modules keep firmware testable and keep MeshCore patch hooks minimal: `MyMesh` observes messages, `BotRuntime` decides, then `MyMesh` sends by existing `sendMessage()`/`sendGroupMessage()`.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/message_handler.py`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/command_manager.py`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not port Python plugins or SQLite/web/API features into firmware. Do not put channel policy inside each command handler; emergency/Public rules must be enforced centrally before execution.
### ITEM-architecture-3: Use strict channel policy: DMs, #bot, #testing, and emergency-only Public forwarding
- **Recommendation:** Default `BotPolicy` should allow normal bot command handling only for DMs, `#bot`, and `#testing`. It should ignore normal `Public` traffic entirely. Messages on `#emergency` should not run the normal command set; instead they should trigger a dedicated emergency-forward path to Public. Store allowlisted channel names as fixed strings or channel indexes resolved at boot, and re-resolve after `CMD_SET_CHANNEL`/channel load.
- **Rationale:** The project explicitly wants normal bot traffic off Public but wants emergency announcements routed to Public. MeshCore group messages are received by channel secret/hash and mapped to `ChannelDetails` in `BaseChatMesh`; `MyMesh` already has `findChannelIdx()` and `getChannel()` to get channel name. Central channel policy prevents a command override from accidentally enabling noisy public replies.
- **Confidence:** HIGH
- **Source:** Project requirements — `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; Local code — `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not copy the Python default `monitor_channels = general,test,emergency`; it contradicts the new scope. Do not use substring channel matching; use exact normalized channel names or resolved channel indexes to avoid `#botnet`/`#testing2` mistakes.
### ITEM-architecture-4: Forward #emergency to Public with a LoRa-native two-stage formatter
- **Recommendation:** Implement `EmergencyForwarder` as a special `BotRuntime` path: when a valid `#emergency` channel message arrives, send to Public as either one packet if it fits (`EMERGENCY MESSAGE FROM <user>: <original text>`) or two ordered Public packets when needed: `EMERGENCY MESSAGE FROM <user>` followed by the original message text truncated to MeshCore’s single-message limit with an explicit truncation marker if necessary. Deduplicate by emergency fingerprint for a short TTL so multiple bots do not all forward the same emergency.
- **Rationale:** `BaseChatMesh::sendGroupMessage()` prepends `<node_name>: ` and clamps total group text to `MAX_TEXT_LEN` (160 bytes in the local source). MeshCore packet payload is capped at `MAX_PACKET_PAYLOAD` 184 bytes. The emergency header consumes payload budget; forcing everything into one packet can silently truncate the important text. A two-stage formatter preserves the required phrase and the original text better while staying LoRa-native. Public is preconfigured in current `MyMesh::begin()`, but saved channel config can override indexes, so Public should be found by name/secret rather than assumed to be index 0.
- **Confidence:** HIGH
- **Source:** Project requirements + Local code — `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/docs/packet_structure.md`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not let `#emergency` execute normal bot commands. Do not forward to Public without deduplication. Do not send unlimited multipart emergency text in v1; each extra packet increases congestion during emergencies.
### ITEM-architecture-5: Fingerprint requests from decrypted semantics, not packet bytes alone
- **Recommendation:** Define `BotMessageFingerprint` as 64 bits from SHA-256 over stable application fields: message kind (DM/channel/emergency), channel name or DM peer key prefix, sender identity where available, sender timestamp, normalized command text, and payload type. Keep RF packet hash/path/SNR as metadata only.
- **Rationale:** MeshCore already deduplicates raw packets, but semantically identical bot requests can arrive through different RF envelopes, and independently generated bot responses will never share a packet hash. The Python bot/coordinator concepts key on logical message identity, while MeshCore packet hash is designed for route duplicate suppression. Firmware needs both: semantic fingerprint for bot suppression and raw packet/path metadata for diagnostics and scoring.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/SimpleMeshTables.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/Mesh.cpp`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/message_handler.py`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not key by text alone; two users can send `ping` simultaneously. Do not key only by `Packet::calculatePacketHash()`; the same command-level event may be represented differently at RF/path layers.
### ITEM-architecture-6: Coordinate with a bounded passive listen-before-answer FSM
- **Recommendation:** Implement `ResponseCoordinator` as fixed-size state tables and a run-to-completion FSM: `OBSERVED -> ELIGIBLE -> PENDING_SELF -> SENT_SELF | SUPPRESSED_BY_KNOWN_BOT | EXPIRED | FAILED`. Events are `message_seen`, `score_ready`, `timer_due`, `known_bot_response_seen`, `known_claim_seen` (future), `send_ok`, `send_failed`, and `ttl_expired`. Default behavior should be passive: compute a score, schedule a response in a 300-3500 ms window, and cancel if a trusted known-bot response/claim for the same fingerprint arrives first.
- **Rationale:** The project explicitly prefers separate bot delays and passive suppression before explicit claim frames. MeshCore’s packet manager already supports scheduled outbound packets, but command execution and suppression should remain app-level so ACK/path behavior is not suppressed. A fixed FSM avoids blocking `loop()`, serial/BLE handling, radio RX, and flash writes.
- **Confidence:** HIGH
- **Source:** Project requirements + Local code — `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/StaticPoolPacketManager.cpp`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not call `delay()` before responding. Do not allocate per-message heap objects. Do not start with mandatory claim/election packets; they may add more airtime than they save.
### ITEM-architecture-7: Trust known bot identities explicitly; do not trust arbitrary group text
- **Recommendation:** Add a persisted `KnownBotRegistry` keyed by full 32-byte public keys plus optional display name and capability flags. Suppression should be authoritative only when the competing bot identity can be validated: DM/control data tied to a known key, signed coordination metadata, or a future signed group-data/claim frame. For plain `GRP_TXT`, treat matching known bot names as a weak hint for low-risk `#bot`/`#testing` only, and never let unauthenticated group text suppress emergency forwarding or safety-critical behavior.
- **Rationale:** MeshCore docs and local code identify group text as unverified: it is encrypted to the channel but the sender name is just text in the group payload. A malicious or mistaken node can spoof `KnownBot: ping`. The user requires trusting known bot identities only, which means the firmware needs explicit key-based trust. Passive suppression can still work for normal channels, but the architecture must not pretend group text alone authenticates a bot.
- **Confidence:** HIGH
- **Source:** Local docs/code — `/Users/cjvana/Documents/GitHub/MeshCore/docs/packet_structure.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/src/Packet.h`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not trust node names, response prefixes, or arbitrary control payloads as identities. Do not use unauthenticated suppression for `#emergency`.
### ITEM-architecture-8: Keep v1 commands small and deterministic
- **Recommendation:** Ship only firmware-suitable commands first: `ping`, `test/t`, `help`, `status`, `about`, `channels`, minimal `path`/routing summary from available packet metadata, and DM-only `advert`. Keep weather, AQI, sports, satpass, web viewer, Discord, SQLite stats, and external API commands out of firmware unless a board-specific host/network adapter is later added.
- **Rationale:** Representative targets include RAK4631/nRF52840 and Heltec v3/ESP32-S3. External data commands in `meshcore-bot` depend on Python, HTTP APIs, local databases, or host services. A firmware-only bot should prioritize reliable off-grid responses, not feature parity. The Python bot remains the behavioral reference for command words, rate limits, channel filtering, and response style.
- **Confidence:** HIGH
- **Source:** Local code/config — `/Users/cjvana/Documents/GitHub/meshcore-bot/config.ini.example`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/message_handler.py`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/commands`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not embed Python, JavaScript, SQLite, TLS HTTP clients, or large plugin registries. Do not claim full `meshcore-bot` command parity in firmware v1.
### ITEM-architecture-9: Persist bot settings in a versioned BotPrefs file and expose protocol plus rescue CLI settings
- **Recommendation:** Add a small versioned `BotPrefs` persisted separately from `NodePrefs`, e.g. `/bot_prefs_v1`, with: enabled/passive, tier, base/max delay, jitter, response cooldowns, allowed channel names/indexes, emergency source/destination channel names, known bot public-key slots, command enable bitset, and stats reset counter. Expose companion protocol commands such as `CMD_GET_BOT_CONFIG`, `CMD_SET_BOT_CONFIG`, `CMD_GET_BOT_STATS`, and `CMD_SET_KNOWN_BOT` in an unused/new range gated by firmware version/capability. Add rescue CLI commands for minimal field recovery: `bot enable|disable`, `bot tier`, `bot channels`, `bot known add <64hex>`, and `bot stats`.
- **Rationale:** Existing `NodePrefs` serialization is manually offset-based; changing it directly risks compatibility mistakes. A separate versioned file isolates bot migration and can be ignored by stock clients. The companion protocol already uses binary commands, async push notifications, response codes, and version/response-length gating; CLI rescue currently supports simple field maintenance when apps are unavailable.
- **Confidence:** HIGH
- **Source:** Local code + official docs — `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/NodePrefs.h`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/DataStore.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; https://docs.meshcore.io/companion_protocol/
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not append fields to `NodePrefs` without a migration/version plan. Do not make all bot config compile-time only; Colorado Mesh deployments need field tuning. Do not require a host app to recover from a bad bot setting.
### ITEM-architecture-10: Design RAM/storage budgets around RAK4631 first, not ESP32 first
- **Recommendation:** Budget the firmware bot for the tight representative target, RAK4631: keep runtime bot RAM under 4 KB in v1 and filesystem bot storage under 4 KB. A concrete target design is: 12 pending coordinator entries at 64 bytes (768 B), 24 recent fingerprint entries at 16 bytes (384 B), 16 known bots at 40 bytes (640 B RAM if loaded; ~640 B persisted), stats/counters under 256 B, and one shared 160-byte response scratch buffer reused from bot runtime. Keep command response strings in flash/PROGMEM where supported.
- **Rationale:** RAK4631 is nRF52840 with 1 MB flash and 256 KB RAM; local linker for RAK companion with extra FS exposes ~712,704 bytes executable flash and ~237,568 bytes RAM after SoftDevice reservation. Heltec v3 has much more flash (8 MB) and ESP32-S3 SRAM (512 KB), so RAK4631 is the limiting design point. Existing companion firmware already reserves large static tables: `MAX_CONTACTS=350`, `MAX_GROUP_CHANNELS=40`, offline queue up to 256 on BLE builds, packet pools, contacts, channels, and advert blobs. Bot state must stay compact.
- **Confidence:** HIGH
- **Source:** Official hardware docs + local code — https://docs.rakwireless.com/product-categories/wisblock/rak4631/overview/ ; https://documentation.espressif.com/esp32-s3_datasheet_en.pdf ; `/Users/cjvana/Documents/GitHub/MeshCore/variants/rak4631/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/boards/nrf52840_s140_v6_extrafs.ld`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.h`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not size the feature using Heltec v3 headroom. Do not add persistent message history, SQLite-like logs, or heap-heavy registries on-device. Do not increase `OFFLINE_QUEUE_SIZE`/packet pool to hide bot backpressure.
### ITEM-architecture-11: Firmware bot flash/storage estimate is feasible if kept to a compact C++ core
- **Recommendation:** Treat v1 firmware bot overhead as feasible but require measurement in CI. Expected incremental resources for the compact design are approximately 20-60 KB flash/rodata, 2-4 KB RAM, and 1-4 KB filesystem storage. Compare that against representative devices: Heltec v3 has 8 MB flash and 512 KB ESP32-S3 SRAM; RAK4631 has 1 MB flash/256 KB RAM, with the local RAK companion executable limit set to 712,704 bytes. Existing v1.15 release assets show Heltec v3 merged images around 695 KB (USB) to 1.33 MB (BLE, includes bootloader/partition gaps) and RAK4631 UF2 assets around 955-971 KB (UF2 encoding, not raw app size); local RAK linker limit remains the useful compile-time constraint.
- **Rationale:** The bot’s proposed static tables are small compared with existing contact/channel/offline queue tables. The bigger risk is flash growth from careless libraries, string-heavy commands, debug logs, or enabling ESP32 WiFi/OTA-style dependencies globally. A compact command set and no dynamic plugin system should fit RAK4631, but exact headroom cannot be asserted until PlatformIO builds produce `.text/.data/.bss` reports for Heltec v3 and RAK4631.
- **Confidence:** MEDIUM
- **Source:** Official release/hardware docs + local config + asset HEAD check — https://github.com/meshcore-dev/MeshCore/releases/tag/companion-v1.15.0 ; https://docs.heltec.cn/en/node/esp32/wifi_lora_32/index.html ; https://docs.rakwireless.com/product-categories/wisblock/rak4631/overview/ ; `/Users/cjvana/Documents/GitHub/MeshCore/variants/heltec_v3/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/variants/rak4631/platformio.ini`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not rely on GitHub asset file size as exact app flash size, especially UF2 and merged ESP32 binaries. Do not add external API clients or large command data tables until actual build reports show headroom.
### ITEM-architecture-12: Use scoring as a delay input, not lower-layer MeshCore timing knobs
- **Recommendation:** Compute a per-message bot score from command eligibility, channel priority, directness/path length, recent heard path, SNR only for direct/zero-hop observations, tier preference, queue health, and deterministic tiebreaker from known bot key/fingerprint. Convert score into a coordinator delay with bounded jitter. Do not repurpose MeshCore `txdelay`, `direct.txdelay`, or `rxdelay` as bot election controls.
- **Rationale:** MeshCore lower-layer timing affects flood/direct retransmit and receive processing, not application-level “which bot answers.” The project already decided to use bot-specific coordinator delays and suppression windows. Keeping this separate avoids breaking path discovery, ACK behavior, and repeater fairness.
- **Confidence:** HIGH
- **Source:** Project requirements + local code — `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/Dispatcher.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/CommonCLI.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/simple_repeater/MyMesh.cpp`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not use topology tier as a fixed “winner.” Do not configure values outside existing MeshCore validation ranges just to influence bot replies.
### ITEM-architecture-13: Instrument duplicate suppression and storage headroom from day one
- **Recommendation:** Add `BotStats` counters and a stats protocol subtype: observed eligible messages, commands executed, emergency forwards, pending scheduled, self sent, suppressed by known response, suppressed by signed claim, suppressed by weak group hint, expired, failed sends, duplicate emergency fingerprints, outbound queue high-water mark, packet pool failures, free heap/stack where available, bot RAM config size, and DataStore used/total KB.
- **Rationale:** The project goal is to reduce duplicate bot traffic without excessive latency. Existing companion firmware already exposes core/radio/packet stats and storage totals. Bot-specific counters are necessary to prove the firmware coordinator helps and to catch false suppression, queue exhaustion, or storage pressure on RAK4631.
- **Confidence:** HIGH
- **Source:** Local code + official docs — `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/DataStore.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/docs/stats_binary_frames.md`; https://docs.meshcore.io/companion_protocol/
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not tune delay windows by anecdote. Do not only count transmissions; false suppression and failed emergency forwarding are higher-risk metrics.
### ITEM-architecture-14: Preserve upstream maintainability with a submodule plus small patch series
- **Recommendation:** Structure the Colorado Mesh project as a wrapper repository with upstream MeshCore pinned as a git submodule (for example `vendor/MeshCore`) plus a versioned patch series (`patches/meshcore/*.patch`) and optional overlay files. CI should initialize the submodule, verify the upstream commit/tag, apply patches cleanly, build representative environments, and fail if patches no longer apply. Keep patches small and upstream-shaped: one for bot source files, one for `MyMesh` hooks/protocol constants, one for DataStore/BotPrefs, one for build flags/CI if needed.
- **Rationale:** The user chose upstream MeshCore as a submodule/patch base. PlatformIO source filters and example paths make a pure out-of-tree plugin hard, so the maintainable compromise is a clean patch stack over upstream rather than a drifting vendored copy. This preserves a clear diff for eventual upstream discussion while allowing Colorado-specific bot policy to stay outside upstream until proven.
- **Confidence:** HIGH
- **Source:** Project requirements + local code — `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/variants/heltec_v3/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/variants/rak4631/platformio.ini`; https://github.com/meshcore-dev/MeshCore/releases/tag/companion-v1.15.0
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not maintain a permanent copy of MeshCore with unstructured edits. Do not create board-specific forks. Do not make the wrapper call into files outside the MeshCore tree if PlatformIO source filters cannot reliably include them across all environments.
## Confidence Summary
| Item ID | Level | Source Type | URL/Reference |
|---------|-------|-------------|---------------|
| ITEM-architecture-1 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/src/Mesh.cpp` |
| ITEM-architecture-2 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/message_handler.py`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/command_manager.py`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp` |
| ITEM-architecture-3 | HIGH | Project requirements + local code | `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp` |
| ITEM-architecture-4 | HIGH | Project requirements + local code docs | `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/docs/packet_structure.md` |
| ITEM-architecture-5 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/SimpleMeshTables.h`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/message_handler.py` |
| ITEM-architecture-6 | HIGH | Project requirements + local code | `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/StaticPoolPacketManager.cpp` |
| ITEM-architecture-7 | HIGH | Local docs/code | `/Users/cjvana/Documents/GitHub/MeshCore/docs/packet_structure.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp` |
| ITEM-architecture-8 | HIGH | Local bot code/config | `/Users/cjvana/Documents/GitHub/meshcore-bot/config.ini.example`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/commands` |
| ITEM-architecture-9 | HIGH | Local code + official docs | https://docs.meshcore.io/companion_protocol/ ; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/DataStore.cpp` |
| ITEM-architecture-10 | HIGH | Hardware docs + local linker/config | https://docs.rakwireless.com/product-categories/wisblock/rak4631/overview/ ; https://documentation.espressif.com/esp32-s3_datasheet_en.pdf ; `/Users/cjvana/Documents/GitHub/MeshCore/boards/nrf52840_s140_v6_extrafs.ld` |
| ITEM-architecture-11 | MEDIUM | Release/hardware docs + local config + asset HEAD check | https://github.com/meshcore-dev/MeshCore/releases/tag/companion-v1.15.0 ; https://docs.heltec.cn/en/node/esp32/wifi_lora_32/index.html ; https://docs.rakwireless.com/product-categories/wisblock/rak4631/overview/ |
| ITEM-architecture-12 | HIGH | Project requirements + local code | `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/Dispatcher.cpp` |
| ITEM-architecture-13 | HIGH | Local code + official docs | `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/DataStore.cpp`; https://docs.meshcore.io/companion_protocol/ |
| ITEM-architecture-14 | HIGH | Project requirements + local code + official release | `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/platformio.ini`; https://github.com/meshcore-dev/MeshCore/releases/tag/companion-v1.15.0 |

214
.forge/research/codex-analysis.md Normal file
View File

@@ -0,0 +1,214 @@
warning: `--full-auto` is deprecated; use `--sandbox workspace-write` instead.
OpenAI Codex v0.130.0
--------
session id: 019e2739-3393-7972-a90b-79f1a369eb12
--------
user
Read .forge/PROJECT.md. Then write a concise research analysis to stdout. Do NOT review any code or plan. Do NOT search the web — analyze based on your training knowledge.
Cover these sections with specific, opinionated recommendations:
1. EXISTING SOLUTIONS — what open-source and commercial products exist in this space
2. RECOMMENDED STACK — specific libraries with versions, and what to avoid
3. ARCHITECTURE — how to structure the system, component boundaries, data flow
4. PITFALLS — domain-specific mistakes and how to prevent them
5. QUESTIONS — what you would ask before building this
Output ONLY your analysis text. No tool call logs, no search results, just the analysis.
2026-05-14T16:02:02.319622Z ERROR rmcp::transport::worker: worker quit with fatal: Transport channel closed, when Auth(TokenRefreshFailed("Server returned error response: invalid_grant: Token refresh failed: invalid or expired grant"))
succeeded in 0ms:
# Forge Project
## Description
Build firmware for MeshCore bots using the standard MeshCore bot project (https://github.com/agessaman/meshcore-bot.git) and MeshCore companion firmware (https://github.com/meshcore-dev/MeshCore.git). The goal is to reduce bot adverts and evaluate/build a built-in response coordinator in firmware as a better alternative to the current VPS coordinator.
## Constraints
- Use the upstream MeshCore bot project and MeshCore companion firmware as primary references.
- Target all MeshCore companion device types where feasible.
- Perform only read-only discovery on local GitHub projects and the hosts `cj-vps` and `cjvana@10.0.0.222` unless later authorized otherwise.
- Deliverable type: code.
## Context
- Greenfield/Brownfield: New project in the Colorado Mesh org, starting from an empty local directory.
- Platform: MeshCore companion firmware / embedded firmware for companion device types.
- Deliverable type: code
- Date: 2026-05-14
## Initial Questions and Answers
- What kind of deliverable should Forge produce? Code project.
- New project or existing local project? New project in the Colorado Mesh org.
- Which target hardware should the first firmware build support? All companion types.
- May local projects and named hosts be inspected? Read-only discovery is OK.
## Candidate Coordination Approach
The user suggested using configured transmit and receive delays to bias which bot responds, potentially approximating which bot is closest or most appropriate for the original message. Proposed topology tiers:
- HILLTOP: highest elevation / backbone infrastructure, typical neighbors 20+, commands `set txdelay 2`, `set direct.txdelay 2`, `set rxdelay 3`.
- FOOTHILLS: mid elevation bridge nodes, typical neighbors 10-20, commands `set txdelay 1.5`, `set direct.txdelay 1`, `set rxdelay 3`.
- SUBURBAN: average rooftop installs, typical neighbors 5-10, commands `set txdelay 0.8`, `set direct.txdelay 0.4`, `set rxdelay 3`.
- LOCAL: low elevation / immediate area, typical neighbors 1-3, commands `set txdelay 0.3`, `set direct.txdelay 0.1`, `set rxdelay 3`.
- MOBILE: variable elevation, should defer to fixed infrastructure, commands `set txdelay 3`, `set direct.txdelay 2.5`, `set rxdelay 3`.
Research and planning should evaluate whether this delay-tier model can reduce duplicate bot adverts/responses, whether it conflicts with MeshCore's existing TX/RX timing semantics, and whether extra coordinator messages are still needed.
codex
1. EXISTING SOLUTIONS
Open source: MeshCore is the correct substrate; stay close to its companion firmware and bot project instead of inventing a parallel mesh layer. Meshtastic is the closest comparable mature LoRa mesh ecosystem and is useful for lessons on channel utilization, node roles, store-and-forward behavior, and UX, but its chat-first model should not be copied wholesale. Reticulum/LXMF is relevant for delay-tolerant addressing and opportunistic links. APRS/LoRa APRS is worth studying for beacon discipline, duplicate suppression, and digipeater etiquette. ChirpStack/LoRaWAN stacks solve a different centralized network-server problem and are mostly a cautionary example here.
Commercial: goTenna Pro, Garmin inReach, ZOLEO, Somewear, Rajant, Silvus, StreamCaster, and Doodle Labs all solve parts of resilient off-grid messaging or tactical mesh. The useful takeaway is not their architecture, which is mostly closed and heavier than MeshCore, but their policy model: role-aware nodes, conservative airtime use, explicit priority, and aggressive duplicate suppression.
Opinionated recommendation: build a distributed soft-coordination layer inside the bot firmware, not a firmware clone of the VPS coordinator. Delay-based response suppression is the right first mechanism because every explicit coordinator frame costs scarce airtime.
2. RECOMMENDED STACK
Use the upstream MeshCore build system and dependency pins wherever they already exist. Do not introduce a new RTOS, packet stack, or language runtime.
Recommended additions if the project needs new code:
- C++17, matching upstream embedded style.
- PlatformIO Core 6.1.x for reproducible multi-board firmware builds if upstream already uses PlatformIO.
- RadioLib 7.1.x only if MeshCore already depends on it or exposes it cleanly; otherwise use MeshCore radio abstractions directly.
- nanopb 0.4.9.1 only for compact structured payloads that must evolve over time.
- ArduinoJson 7.2.x only for local config import/export or serial diagnostics, not over the air.
- Unity 2.6.x for embedded unit tests.
- Python 3.12, pytest 8.3.x, Hypothesis 6.112.x, and SimPy 4.1.x for host-side coordination and airtime simulations.
Avoid: JSON over LoRa, MQTT in firmware, full protobuf runtimes, SQLite, dynamic plugin systems, heap-heavy callback code, custom radio drivers, and any design that requires every bot to hear every other bot.
3. ARCHITECTURE
Structure the firmware as a bot coordination module sitting above MeshCore transport:
RX path: MeshCore packet -> bot request classifier -> request ID normalization -> eligibility engine -> response scheduler.
TX path: pending response intent -> suppression window -> overheard-response cancellation -> final response publish -> seen/sent ring buffer update.
Core components:
- Request classifier: identifies bot-triggering messages and derives a stable idempotency key from origin, command type, normalized content, and a short time bucket.
- Eligibility engine: decides whether this bot may answer based on role, capability, directness, recent neighbor observations, and command type.
- Scheduler: computes `response_at = rx_time + role_delay + airtime_scaled_jitter`.
- Suppression cache: cancels pending responses when an equivalent valid response is heard.
- Metrics: counts eligible, deferred, canceled, transmitted, duplicate-heard, and hidden-node-suspected events.
- Config: expose role, base delay, jitter, suppression window, and command-specific overrides through the existing MeshCore CLI/config style.
The proposed tier model is directionally good: LOCAL and SUBURBAN should usually answer before HILLTOP and MOBILE. Treat the tiers as policy priority, not true distance. Do not overload MeshCore's core `txdelay` semantics if those affect all radio traffic; add bot-specific coordination delays instead. The suppression window should be longer than the slowest role delay plus worst-case packet airtime plus jitter. A universal `rxdelay 3` is likely too short if MOBILE can delay 3 seconds.
Do not add coordinator claim frames in the first version. Measure duplicate rate first. If duplicates remain unacceptable, add a very small authenticated "claim" frame only for expensive or high-noise responses.
4. PITFALLS
Delay tiers will not solve hidden nodes. Two bots may both hear the requester but not each other. Prevent damage with idempotency keys, client-side duplicate tolerance, and metrics that reveal hidden-node cases.
Neighbor count is not proximity. Hilltop nodes may hear many nodes but be the wrong responder for local context. Use configured role first, observed directness second, RSSI/SNR only as a weak tie-breaker.
Airtime math matters. LoRa packet airtime can dominate small delay differences, especially at slow spreading factors. Jitter should scale with estimated airtime, not be a fixed tiny random number.
Do not run logic inside radio callbacks. Callbacks should enqueue compact events; normal firmware loop/task code should parse, schedule, and transmit.
Do not persist every seen request. Use RAM ring buffers for hot dedupe and write config/metrics sparingly to avoid flash wear.
Do not trust unauthenticated suppression. A forged response or claim could silence all bots. Reuse MeshCore identity/authentication primitives where available.
Do not assume all companion device types have equal RAM, flash, display, BLE, or radio behavior. Keep the coordinator core platform-neutral and put board-specific behavior behind existing MeshCore abstractions.
5. QUESTIONS
What exact traffic should be suppressed: bot adverts, command replies, telemetry, direct-message responses, or all of them?
What duplicate rate and response latency are acceptable in the field?
Should the system prefer the nearest/local bot, the most reliable fixed bot, or the least airtime-costly bot?
How does the current VPS coordinator decide winners, and what failures are driving the firmware replacement?
Are role tiers manually configured, automatically inferred, or both?
Which companion device is the minimum hardware baseline for RAM, flash, and radio capability?
Can packet formats change, or must this remain fully compatible with stock MeshCore nodes?
Are adversarial or misconfigured nodes in scope for the security model?
What RF settings, region, channel usage, and typical packet sizes should simulations target?
Should emergency/admin commands bypass normal delay suppression?
22,375
1. EXISTING SOLUTIONS
Open source: MeshCore is the correct substrate; stay close to its companion firmware and bot project instead of inventing a parallel mesh layer. Meshtastic is the closest comparable mature LoRa mesh ecosystem and is useful for lessons on channel utilization, node roles, store-and-forward behavior, and UX, but its chat-first model should not be copied wholesale. Reticulum/LXMF is relevant for delay-tolerant addressing and opportunistic links. APRS/LoRa APRS is worth studying for beacon discipline, duplicate suppression, and digipeater etiquette. ChirpStack/LoRaWAN stacks solve a different centralized network-server problem and are mostly a cautionary example here.
Commercial: goTenna Pro, Garmin inReach, ZOLEO, Somewear, Rajant, Silvus, StreamCaster, and Doodle Labs all solve parts of resilient off-grid messaging or tactical mesh. The useful takeaway is not their architecture, which is mostly closed and heavier than MeshCore, but their policy model: role-aware nodes, conservative airtime use, explicit priority, and aggressive duplicate suppression.
Opinionated recommendation: build a distributed soft-coordination layer inside the bot firmware, not a firmware clone of the VPS coordinator. Delay-based response suppression is the right first mechanism because every explicit coordinator frame costs scarce airtime.
2. RECOMMENDED STACK
Use the upstream MeshCore build system and dependency pins wherever they already exist. Do not introduce a new RTOS, packet stack, or language runtime.
Recommended additions if the project needs new code:
- C++17, matching upstream embedded style.
- PlatformIO Core 6.1.x for reproducible multi-board firmware builds if upstream already uses PlatformIO.
- RadioLib 7.1.x only if MeshCore already depends on it or exposes it cleanly; otherwise use MeshCore radio abstractions directly.
- nanopb 0.4.9.1 only for compact structured payloads that must evolve over time.
- ArduinoJson 7.2.x only for local config import/export or serial diagnostics, not over the air.
- Unity 2.6.x for embedded unit tests.
- Python 3.12, pytest 8.3.x, Hypothesis 6.112.x, and SimPy 4.1.x for host-side coordination and airtime simulations.
Avoid: JSON over LoRa, MQTT in firmware, full protobuf runtimes, SQLite, dynamic plugin systems, heap-heavy callback code, custom radio drivers, and any design that requires every bot to hear every other bot.
3. ARCHITECTURE
Structure the firmware as a bot coordination module sitting above MeshCore transport:
RX path: MeshCore packet -> bot request classifier -> request ID normalization -> eligibility engine -> response scheduler.
TX path: pending response intent -> suppression window -> overheard-response cancellation -> final response publish -> seen/sent ring buffer update.
Core components:
- Request classifier: identifies bot-triggering messages and derives a stable idempotency key from origin, command type, normalized content, and a short time bucket.
- Eligibility engine: decides whether this bot may answer based on role, capability, directness, recent neighbor observations, and command type.
- Scheduler: computes `response_at = rx_time + role_delay + airtime_scaled_jitter`.
- Suppression cache: cancels pending responses when an equivalent valid response is heard.
- Metrics: counts eligible, deferred, canceled, transmitted, duplicate-heard, and hidden-node-suspected events.
- Config: expose role, base delay, jitter, suppression window, and command-specific overrides through the existing MeshCore CLI/config style.
The proposed tier model is directionally good: LOCAL and SUBURBAN should usually answer before HILLTOP and MOBILE. Treat the tiers as policy priority, not true distance. Do not overload MeshCore's core `txdelay` semantics if those affect all radio traffic; add bot-specific coordination delays instead. The suppression window should be longer than the slowest role delay plus worst-case packet airtime plus jitter. A universal `rxdelay 3` is likely too short if MOBILE can delay 3 seconds.
Do not add coordinator claim frames in the first version. Measure duplicate rate first. If duplicates remain unacceptable, add a very small authenticated "claim" frame only for expensive or high-noise responses.
4. PITFALLS
Delay tiers will not solve hidden nodes. Two bots may both hear the requester but not each other. Prevent damage with idempotency keys, client-side duplicate tolerance, and metrics that reveal hidden-node cases.
Neighbor count is not proximity. Hilltop nodes may hear many nodes but be the wrong responder for local context. Use configured role first, observed directness second, RSSI/SNR only as a weak tie-breaker.
Airtime math matters. LoRa packet airtime can dominate small delay differences, especially at slow spreading factors. Jitter should scale with estimated airtime, not be a fixed tiny random number.
Do not run logic inside radio callbacks. Callbacks should enqueue compact events; normal firmware loop/task code should parse, schedule, and transmit.
Do not persist every seen request. Use RAM ring buffers for hot dedupe and write config/metrics sparingly to avoid flash wear.
Do not trust unauthenticated suppression. A forged response or claim could silence all bots. Reuse MeshCore identity/authentication primitives where available.
Do not assume all companion device types have equal RAM, flash, display, BLE, or radio behavior. Keep the coordinator core platform-neutral and put board-specific behavior behind existing MeshCore abstractions.
5. QUESTIONS
What exact traffic should be suppressed: bot adverts, command replies, telemetry, direct-message responses, or all of them?
What duplicate rate and response latency are acceptable in the field?
Should the system prefer the nearest/local bot, the most reliable fixed bot, or the least airtime-costly bot?
How does the current VPS coordinator decide winners, and what failures are driving the firmware replacement?
Are role tiers manually configured, automatically inferred, or both?
Which companion device is the minimum hardware baseline for RAM, flash, and radio capability?
Can packet formats change, or must this remain fully compatible with stock MeshCore nodes?
Are adversarial or misconfigured nodes in scope for the security model?
What RF settings, region, channel usage, and typical packet sizes should simulations target?
Should emergency/admin commands bypass normal delay suppression?

235
.forge/research/pitfalls.md Normal file
View File

@@ -0,0 +1,235 @@
# Pitfalls Research: Firmware-only MeshCore Bot
Project: firmware-only embedded bot command handling in MeshCore companion firmware. Upstream MeshCore is the firmware base/submodule/patch base; `meshcore-bot` is a behavioral reference only. Normal bot traffic is limited to private DMs, `#bot`, and `#testing`; `#emergency` is forwarded to Public as an emergency announcement. Representative build targets: Heltec v3 and RAK4631.
Checked: 2026-05-14
Storage baseline used in this research:
| Target | Hardware/storage facts | Current firmware size signal | Headroom implication |
|---|---|---|---|
| Heltec WiFi LoRa 32 V3 | ESP32-S3FN8, 8 MB SiP flash, 512 KB SRAM, no external PSRAM, SX1262 | MeshCore v1.15.0 release assets: Heltec v3 USB app `.bin` 615 KB; BLE app `.bin` 1.2 MB; merged BLE 1.27 MB | Flash is comfortable for a small firmware bot, but RAM is still constrained and BLE/display/WiFi variants can consume much more than USB. |
| RAK4631 | nRF52840, 1 MB flash, 256 KB RAM, BLE 5.0, SX1262; local companion env caps app at 712,704 bytes and RAM region at 237,568 bytes after SoftDevice reservation | MeshCore v1.15.0 release assets: RAK4631 USB `.uf2` 933 KB / `.zip` 467 KB; BLE `.uf2` 949 KB / `.zip` 475 KB. UF2 includes container overhead, so zip size is the best visible proxy without a local ELF/map build. | RAK4631 is the limiting target. A compact parser/coordinator is likely feasible; Python-bot parity or large lookup tables are not. |
Estimated firmware-only bot budget, before measurement: a compact built-in bot should target **20-60 KB additional flash**, **2-8 KB static/RAM**, **0 heap allocation after setup**, and **<1 KB persisted prefs**. A fuller command suite with many string templates, JSON/API logic, large help text, or display telemetry can easily exceed this and should be gated per target. These estimates are MEDIUM confidence because no local PlatformIO build/map was available in this repo.
### ITEM-pitfalls-1: Underestimating RAK4631 as the storage/RAM limiting device
- **What goes wrong:** The bot fits and behaves on Heltec v3 ESP32-S3, then fails to link, crashes, or becomes unstable on RAK4631 BLE because the nRF52840 target has far less usable flash/RAM once SoftDevice, BLE, display, contact tables, channel tables, offline queue, packet pools, and filesystem regions are included.
- **Root cause:** Heltec v3 has 8 MB flash and 512 KB SRAM, while RAK4631 has 1 MB flash and 256 KB RAM. The local RAK4631 companion env further sets `board_upload.maximum_size = 712704`, and its linker script places application flash from `0x26000` to `0xD4000` with RAM from `0x20006000` to `0x20040000`. Current upstream release assets show RAK4631 companion firmware is already hundreds of KB before bot code, and the BLE variant carries additional queues/logging/display code.
- **Prevention:** Make RAK4631 BLE the hard budget target, not Heltec v3. Require every bot feature PR to report `.text/.rodata/.data/.bss` deltas from PlatformIO map/size output for `RAK_4631_companion_radio_ble`, `RAK_4631_companion_radio_usb`, `Heltec_v3_companion_radio_ble`, and `Heltec_v3_companion_radio_usb`. Keep Phase 1 to a compact command parser, passive duplicate suppression, emergency forwarding, and a few single-packet commands. Defer weather/AQI/sports/satpass/feed/web features unless a build map proves headroom.
- **Severity:** CRITICAL
- **Phase relevance:** Phase 0 feasibility and every CI build gate.
- **Confidence:** HIGH
- **Source:** Local code + official hardware docs + GitHub release assets — `/Users/cjvana/Documents/GitHub/MeshCore/variants/rak4631/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/boards/nrf52840_s140_v6_extrafs.ld`; https://heltec.org/project/wifi-lora-32-v3/ ; https://docs.rakwireless.com/product-categories/wisblock/rak4631/overview/ ; https://github.com/meshcore-dev/MeshCore/releases/expanded_assets/companion-v1.15.0
- **Checked:** 2026-05-14
### ITEM-pitfalls-2: Treating the Python bot as portable command code instead of a behavioral oracle
- **What goes wrong:** Firmware grows into an unmaintainable partial Python-bot clone: many commands require internet APIs, JSON parsing, caching, date/time libraries, HTTP/TLS, Discord/web hooks, SQLite-like history, or large text responses. The result exceeds MCU flash/RAM, cannot work off-grid, and behaves differently from the host bot anyway.
- **Root cause:** The Python bot depends on host resources and libraries (`requests`, `aiohttp`, Flask, feedparser, PyEphem, Open-Meteo clients, cryptography, `meshcore-cli`) and includes command modules for weather, AQI, satellite passes, feeds, sports, web viewer, Discord forwarding, stats databases, and contact management. Those are not firmware-native features.
- **Prevention:** Define a firmware-only minimum viable command set: `ping`, `help` with short static text, status/battery/storage, path/contact diagnostics from existing MeshCore state, and emergency forwarding. Treat Python modules as golden behavior for command names, channel policy, cooldowns, and output style only. Anything needing IP services should stay host-side or be replaced by a cached/static firmware-safe answer.
- **Severity:** CRITICAL
- **Phase relevance:** Scope definition and feature triage.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/meshcore-bot/requirements.txt`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/commands/*`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/message_handler.py`
- **Checked:** 2026-05-14
### ITEM-pitfalls-3: Heap fragmentation from `String`, dynamic containers, JSON, and variable response construction
- **What goes wrong:** The bot passes bench tests but reboots or fails to allocate packets after hours/days of varied commands because small MCU heaps fragment. Failures look like random packet-pool exhaustion, BLE instability, or corrupted command output.
- **Root cause:** Firmware bot parsing tempts use of Arduino `String`, `std::vector`, maps, JSON builders, formatted heap strings, and per-message dynamic objects. Upstream MeshCore already warns contributors to avoid dynamic allocation except during setup/begin, and its packet manager uses fixed pools. The companion currently has only one explicit large heap path for signing (`malloc(MAX_SIGN_DATA_LEN)`), which should not become a model for bot logic.
- **Prevention:** Use fixed-size structs, ring buffers, and `char[]` parsing. Allocate coordinator tables statically; no `malloc/new/String` in receive, parse, schedule, or send paths. Keep command output templates in `const`/flash-friendly storage where supported, and always bound-copy into `MAX_TEXT_LEN`/`MAX_FRAME_SIZE` buffers. Add a static-analysis/code-review rule that bot files cannot use `String`, heap containers, or heap allocation after setup.
- **Severity:** CRITICAL
- **Phase relevance:** Firmware implementation standards and review checklist.
- **Confidence:** HIGH
- **Source:** Local code + ecosystem reference — `/Users/cjvana/Documents/GitHub/MeshCore/README.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/StaticPoolPacketManager.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; https://cpp4arduino.com/2018/11/06/what-is-heap-fragmentation.html
- **Checked:** 2026-05-14
### ITEM-pitfalls-4: Packet-pool and offline-queue exhaustion during delayed bot responses
- **What goes wrong:** A burst of commands creates many pending responses or emergency forwards. The 16-packet pool fills, outbound/inbound queues fill, MeshCore reports table-full/send failures, and normal ACK/path/adverts/routing maintenance are starved. Losing bot responses may still occupy packet slots until their timers fire.
- **Root cause:** Companion `MyMesh` constructs `StaticPoolPacketManager(16)`, and `StaticPoolPacketManager` uses fixed send/rx queues. BLE variants may set `OFFLINE_QUEUE_SIZE=256`, which is a large RAM commitment, while USB defaults to 16. Existing code drops oldest channel messages when offline queue is full, so a firmware bot can accidentally evict user-visible messages with its own generated traffic.
- **Prevention:** Keep pending-response state separate from allocated MeshCore packets. Do not call `createGroupDatagram()` until the response timer actually wins. Limit pending bot windows to a small fixed number, e.g. 8-16. If over capacity, drop low-priority bot replies before user messages. Emergency forwarding may preempt normal bot responses but must still respect packet-pool availability and report a dropped-forward counter.
- **Severity:** CRITICAL
- **Phase relevance:** Coordinator implementation and stress testing.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/StaticPoolPacketManager.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/variants/heltec_v3/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/variants/rak4631/platformio.ini`
- **Checked:** 2026-05-14
### ITEM-pitfalls-5: Exceeding MeshCore text/frame limits with Python-style bot replies
- **What goes wrong:** Help, weather, emergency, sports, stats, or path responses are truncated, rejected, split by future code into multiple packets, or silently lose context. Duplicate suppression may suppress only the first fragment while later fragments still transmit.
- **Root cause:** MeshCore constants are small: `MAX_PACKET_PAYLOAD=184`, `MAX_FRAME_SIZE=172`, and `MAX_TEXT_LEN=160`. Companion protocol docs currently state datagram payload caps around 163 bytes and text messages around 133 characters. `BaseChatMesh::sendGroupMessage()` prefixes group messages with `"<sender>: "`, reducing available bot text. LoRa airtime makes multi-packet bot chatter expensive.
- **Prevention:** Design every firmware command as a single-packet, LoRa-native response. Enforce command-specific output caps before send; include a test that serializes exact channel output with the node name prefix. `#emergency` forwarding must be short enough to fit or use a deterministic two-message policy with bounded truncation: announcement plus truncated original text. Do not implement multipart public bot replies in Phase 1.
- **Severity:** MODERATE
- **Phase relevance:** UX copy, command implementation, emergency forwarding.
- **Confidence:** HIGH
- **Source:** Local code + official docs — `/Users/cjvana/Documents/GitHub/MeshCore/src/MeshCore.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseSerialInterface.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.h`; https://docs.meshcore.io/companion_protocol/
- **Checked:** 2026-05-14
### ITEM-pitfalls-6: Unsafe C++ command parsing and text normalization
- **What goes wrong:** A malformed frame or odd user message causes buffer over-read, unterminated strings, command misfire, or incorrect sender extraction. Inputs containing colons, empty names, control characters, UTF-8, long aliases, or embedded NULs may bypass filters or corrupt output.
- **Root cause:** Existing companion code commonly null-terminates `cmd_frame[len]`, uses `strlen`, `strcpy`, `sprintf`, and colon parsing conventions (`sender: message`) inherited from group text format. Python `meshcore-bot` sanitizes input with helper utilities, but firmware C++ must do this manually under smaller buffers.
- **Prevention:** Build a tiny parser that operates on `(const char*, length)` not untrusted `strlen` until after explicit NUL insertion inside a known spare byte. Strip C0 controls except newline-equivalent spaces; normalize only ASCII command prefixes in Phase 1; treat everything after the command verb as bounded opaque text. Use `snprintf`/bounded copy everywhere. Add fuzz/unit tests for maximum-length messages, no colon, leading colon, repeated colon, UTF-8, NUL, and all allowed channel names.
- **Severity:** CRITICAL
- **Phase relevance:** Command parser implementation and test harness.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/security_utils.py`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/message_handler.py`
- **Checked:** 2026-05-14
### ITEM-pitfalls-7: Misidentifying channels by index instead of stable channel policy
- **What goes wrong:** The bot answers on Public during normal operation, ignores `#bot`, mishandles `#testing`, or forwards the wrong channel because channel indexes differ across devices. A user renames/reorders channels and the bot policy silently changes.
- **Root cause:** Companion receive frames identify channel by `channel_idx`; `BaseChatMesh` stores `ChannelDetails` in an array and `findChannelIdx()` matches channel secrets. Public is auto-added first, then persisted channels load. Indexes are local configuration, not global semantic names. MeshCore channel messages are group-key based and unverified by sender identity at the spec level.
- **Prevention:** Resolve policy by channel secret/hash plus configured display name at startup, then cache explicit channel slots for `Public`, `#bot`, `#testing`, and `#emergency`. Refuse to enable the bot if required channels are ambiguous, missing, or duplicate. Normal responses must be allowed only in DMs, `#bot`, and `#testing`; Public sends are allowed only from the emergency forwarder and self-tests. Expose a diagnostic command showing channel index/name/hash/policy.
- **Severity:** CRITICAL
- **Phase relevance:** Channel setup and command routing.
- **Confidence:** HIGH
- **Source:** Local code + official docs — `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/ChannelDetails.h`; https://docs.meshcore.io/companion_protocol/
- **Checked:** 2026-05-14
### ITEM-pitfalls-8: Emergency forwarding loops, amplification, and spoofed urgency
- **What goes wrong:** A message in `#emergency` is forwarded repeatedly by every bot to Public; bots then see the Public announcement and re-forward or respond; malicious users trigger panic banners; long emergency text consumes multiple high-priority packets during congestion.
- **Root cause:** The new rule intentionally bridges `#emergency` to Public. Without an idempotency key, TTL, and bot-origin detection, every firmware bot can act on the same emergency packet independently. Group messages identify the text sender string but are not a strong authenticated identity. Passive duplicate suppression without claim frames allows hidden bots to miss each other.
- **Prevention:** Treat emergency forwarding as a separate high-priority, idempotent action keyed by original channel hash, timestamp, normalized original text, and sender string/public key prefix where available. Keep an emergency-forward cache longer than normal bot duplicate windows, e.g. 10-30 minutes. Never forward bot-originated `EMERGENCY MESSAGE FROM` text. Use exactly the user-requested Public format but clamp length. Prefer one winner via the same passive suppression window, but if duplicates occur they must have identical text and no recursive trigger.
- **Severity:** CRITICAL
- **Phase relevance:** Emergency feature design before public testing.
- **Confidence:** HIGH
- **Source:** Project requirement + local code — `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/message_handler.py`
- **Checked:** 2026-05-14
### ITEM-pitfalls-9: Passive duplicate suppression without claim frames cannot guarantee single response
- **What goes wrong:** Two bots that cannot hear each other both answer after their local delay window. Bench tests with colocated radios look clean, but field deployments still produce duplicates across terrain, hidden nodes, and asymmetric paths.
- **Root cause:** Phase 1 intentionally defers explicit on-air claim frames. LoRa listen-before-talk/CAD reduces collisions but does not solve hidden-node consensus. MeshCore’s duplicate tables suppress identical packets, not independently generated bot responses. A final response can only suppress peers that hear it before their timer fires.
- **Prevention:** Set expectations: Phase 1 reduces noise, not guarantees exactly once. Use deterministic scoring plus jitter, short bounded windows, and cancel-on-heard-response. Record duplicate fingerprints in telemetry. Do not add claim frames until field captures prove passive suppression is insufficient; if claims are added, authenticate them and keep them zero-hop/scoped.
- **Severity:** MODERATE
- **Phase relevance:** Coordination algorithm and acceptance criteria.
- **Confidence:** HIGH
- **Source:** Local code + ecosystem docs/search — `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/SimpleMeshTables.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/Mesh.cpp`; https://meshtastic.org/docs/overview/mesh-algo/ ; https://scholarworks.gnu.ac.kr/item/ccb29254-804b-413d-adc5-853fe697241b
- **Checked:** 2026-05-14
### ITEM-pitfalls-10: Suppression keyed by raw packet hash or plain text creates false positives/negatives
- **What goes wrong:** Bots fail to suppress duplicate responses for the same command observed through different RF envelopes, or they suppress unrelated commands because two users typed the same text. Emergency forwarding may drop a real second emergency because the text happens to match.
- **Root cause:** MeshCore flood/direct paths, transport codes, and retry/ACK behavior can change packet bytes while the application-level command is the same. Conversely, text-only matching ignores sender, channel, timestamp, payload type, and DM vs channel semantics.
- **Prevention:** Define a firmware `BotMessageFingerprint` over stable application fields after decryption/parsing: payload kind, channel identity or DM peer identity, sender timestamp, normalized command text, sender public-key prefix/name when available, and a small packet hash tie-in for diagnostics. Store both request fingerprint and response fingerprint. Use a separate emergency fingerprint with a longer TTL.
- **Severity:** CRITICAL
- **Phase relevance:** Coordinator state model and test vectors.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/MeshCore/src/Packet.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/SimpleMeshTables.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/message_handler.py`
- **Checked:** 2026-05-14
### ITEM-pitfalls-11: Trusting spoofable bot identities, response text, or control packets
- **What goes wrong:** A non-bot node suppresses real bots by sending fake claim/control data or a message that looks like a bot response. In emergency mode this can block forwarding or inject false Public emergency announcements.
- **Root cause:** MeshCore `PAYLOAD_TYPE_CONTROL` is a generic control/discovery packet path and companion currently forwards control data to the app without authentication. Group text is shared-key encrypted but the visible `name: message` prefix is not enough to prove a known bot identity. The project decision says suppression should trust known bot identities only.
- **Prevention:** In Phase 1, suppress only on final responses that match known bot public-key prefixes or an allowlisted bot identity table; do not suppress emergency forwarding based on unauthenticated claims. If control/claim packets are introduced later, include version/length/magic/fingerprint and require signature or known-key verification before they can suppress. Parse unknown control frames as untrusted hints or ignore them.
- **Severity:** CRITICAL
- **Phase relevance:** Security design and duplicate suppression.
- **Confidence:** HIGH
- **Source:** Local code/docs — `/Users/cjvana/Documents/GitHub/MeshCore/docs/payloads.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/Packet.h`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`
- **Checked:** 2026-05-14
### ITEM-pitfalls-12: Blocking the companion superloop with bot delays or slow display/BLE/serial work
- **What goes wrong:** While a bot waits to see if another node responds, BLE notifications stall, serial frames time out, display/UI feels frozen, RTC ticks lag, sensors are skipped, or radio RX/TX processing is delayed. The coordinator makes the node less reliable than stock companion firmware.
- **Root cause:** Companion firmware is a cooperative loop: `the_mesh.loop()`, serial/BLE/WiFi interface polling, sensors, display task, and RTC tick all depend on returning quickly. Protocol docs require one in-flight command and handling unsolicited notifications, while local display/UI code also updates from message paths.
- **Prevention:** Implement response delays as scheduled timestamps in a small FSM checked from `MyMesh::loop()`. Never use `delay()` or busy waits for bot coordination. Do not render long bot logs on display; only signal a compact bot event/counter. Respect `_serial->isWriteBusy()` before pushing optional stats. Emergency forwarding can preempt normal bot work but must still be non-blocking.
- **Severity:** CRITICAL
- **Phase relevance:** Main-loop integration and UI/BLE/serial testing.
- **Confidence:** HIGH
- **Source:** Local code + official docs — `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/main.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; https://docs.meshcore.io/companion_protocol/
- **Checked:** 2026-05-14
### ITEM-pitfalls-13: Breaking BLE/serial companion protocol compatibility with bot notifications
- **What goes wrong:** Existing apps and `meshcore-bot` clients misparse frames, lose messages, or deadlock because firmware emits new bot frames while a command response is expected, changes existing response codes, or overflows BLE MTU assumptions.
- **Root cause:** Companion protocol clients are instructed to send one command at a time, wait for a response, and also handle asynchronous notifications. Firmware version/capability fields vary by app target version. Current `MyMesh` has fixed command/response/push code ranges and a local `FIRMWARE_VER_CODE=8` in the read checkout.
- **Prevention:** Do not change existing frame semantics for stock commands. In firmware-only bot mode, prefer no new host-visible frames in Phase 1 except stats/capability behind explicit query. If new frames are required, park them in a new negotiated range, gate by firmware version/capability, keep every frame <= `MAX_FRAME_SIZE`, and ensure old clients safely ignore or never see them.
- **Severity:** MODERATE
- **Phase relevance:** Firmware API and compatibility testing.
- **Confidence:** HIGH
- **Source:** Local code + official docs — `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.h`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; https://docs.meshcore.io/companion_protocol/
- **Checked:** 2026-05-14
### ITEM-pitfalls-14: Misusing SNR/RSSI/path data to choose the “closest” bot
- **What goes wrong:** A bot with a strong last-hop signal wins even though it is not closest to the original sender, or a bot suppresses itself because a stale contact path suggests another node is better. Mobile or asymmetric links make the scoring erratic.
- **Root cause:** MeshCore path fields differ by route type. Flood path is historical route; direct path is routing instructions; TRACE path fields can contain SNR data. RSSI/SNR from the radio is for the last received hop, not end-to-end quality. Python bot code already carries complex RF correlation fallbacks because host-side events can be difficult to match.
- **Prevention:** Use SNR/RSSI only as a weak tie-breaker for zero-hop/direct observations. Primary eligibility should be command support, channel policy, known bot identity, recent directness/path length, queue health, and deterministic bot-id jitter. Keep topology tier as a small bias, not a winner-takes-all delay. Include test captures for multi-hop and TRACE packets.
- **Severity:** MODERATE
- **Phase relevance:** Scoring model and field validation.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/MeshCore/src/Mesh.h`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/message_handler.py`
- **Checked:** 2026-05-14
### ITEM-pitfalls-15: Persisting high-churn bot state and wearing or filling filesystem storage
- **What goes wrong:** Duplicate windows, recent-response caches, bot logs, command history, or emergency history are written to flash repeatedly. Filesystem wear, corruption, or storage-full behavior then breaks identity, contacts, channels, and advert blobs.
- **Root cause:** Companion `DataStore` already persists identity, prefs, contacts, channels, and advert blobs using SPIFFS/LittleFS/InternalFS/ExtraFS. nRF52 targets preallocate advert blob records and may migrate contacts/channels to secondary FS. `CMD_GET_BATT_AND_STORAGE` reports storage but bot runtime state does not need persistence.
- **Prevention:** Persist only bot configuration: enabled flag, tier, channel policy hashes, known bot identity prefixes, cooldown settings, and emergency-forward enabled. Keep all request fingerprints, suppression windows, emergency recent cache, and counters in RAM with bounded TTL. If counters must survive reboot, save coarse totals lazily and rarely, never per message.
- **Severity:** MODERATE
- **Phase relevance:** Persistence design and DataStore changes.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/DataStore.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/DataStore.h`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/NodePrefs.h`
- **Checked:** 2026-05-14
### ITEM-pitfalls-16: NodePrefs binary-layout migration breaks existing companion settings
- **What goes wrong:** Adding bot fields to `NodePrefs` changes struct layout expectations, causing old `/new_prefs` files to load wrong values for radio, BLE PIN, location policy, or buzzer settings. A field-unit mismatch could put bots on the wrong LoRa params or expose a stale BLE PIN.
- **Root cause:** `DataStore::loadPrefsInt()` and `savePrefs()` manually read/write specific offsets into `/new_prefs`; the persisted file is not a self-describing schema. App/device compatibility also depends on existing command frames for tuning and other params.
- **Prevention:** Do not insert fields into the middle of `NodePrefs` or change existing offsets. Add a separate bot prefs file with magic/version/length/CRC, or append only with explicit backwards-compatible load defaults. On first boot after update, validate radio params, channel policy, and bot enable flag; default bot disabled if config is invalid.
- **Severity:** CRITICAL
- **Phase relevance:** Configuration persistence implementation.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/NodePrefs.h`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/DataStore.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`
- **Checked:** 2026-05-14
### ITEM-pitfalls-17: All-companion portability breaks through target-specific assumptions
- **What goes wrong:** Firmware bot code compiles for Heltec v3 but fails on nRF52, RP2040, STM32, USB-only, BLE, WiFi, or no-display companion variants. A command uses ESP32-only APIs, assumes SPIFFS paths, assumes BLE exists, or relies on display/UI classes not present on every target.
- **Root cause:** MeshCore supports many PlatformIO environments with variant overlays and different source filters. Companion `main.cpp` selects filesystem and serial interfaces by platform macros. Heltec v3 has USB/BLE/WiFi variants; RAK4631 has USB/BLE, display, sensors, ExtraFS, and nRF52 SoftDevice constraints.
- **Prevention:** Keep bot logic in platform-neutral companion C++ with no direct ESP32/nRF calls. Put target-specific storage, display, BLE, and WiFi behavior behind existing `MyMesh`, `DataStore`, and serial-interface abstractions. CI must build at least Heltec v3 USB/BLE and RAK4631 USB/BLE before any feature is considered portable; later expand to all companion suffixes.
- **Severity:** CRITICAL
- **Phase relevance:** Build system and portability implementation.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/MeshCore/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/main.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/variants/heltec_v3/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/variants/rak4631/platformio.ini`
- **Checked:** 2026-05-14
### ITEM-pitfalls-18: Private key export/import remains enabled in bot firmware builds
- **What goes wrong:** A deployed bot node exposes identity private-key export/import over the companion protocol, increasing damage from a compromised host, BLE session, or physical access. Known-bot trust and suppression can then be subverted by cloned identities.
- **Root cause:** The upstream base build flags in the local checkout define `ENABLE_PRIVATE_KEY_IMPORT=1` and `ENABLE_PRIVATE_KEY_EXPORT=1` with a comment warning to disable them for more secure firmware. Bot identity becomes more valuable once other nodes trust known bot public keys for suppression and emergency behavior.
- **Prevention:** Disable private key export/import in production bot firmware unless a deliberate provisioning workflow requires it. If identity migration is needed, provide a separate provisioning build or physical-button-gated window. Known-bot identity lists should assume keys are long-lived and protected.
- **Severity:** CRITICAL
- **Phase relevance:** Release build flags and provisioning.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/MeshCore/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`
- **Checked:** 2026-05-14
## Confidence Summary
| Item ID | Level | Source Type | URL/Reference |
|---------|-------|-------------|---------------|
| ITEM-pitfalls-1 | HIGH | Local code + official hardware docs + release assets | `/Users/cjvana/Documents/GitHub/MeshCore/variants/rak4631/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/boards/nrf52840_s140_v6_extrafs.ld`; https://heltec.org/project/wifi-lora-32-v3/ ; https://docs.rakwireless.com/product-categories/wisblock/rak4631/overview/ ; https://github.com/meshcore-dev/MeshCore/releases/expanded_assets/companion-v1.15.0 |
| ITEM-pitfalls-2 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/meshcore-bot/requirements.txt`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/commands/*` |
| ITEM-pitfalls-3 | HIGH | Local code + WebSearch | `/Users/cjvana/Documents/GitHub/MeshCore/README.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/StaticPoolPacketManager.cpp`; https://cpp4arduino.com/2018/11/06/what-is-heap-fragmentation.html |
| ITEM-pitfalls-4 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/StaticPoolPacketManager.cpp` |
| ITEM-pitfalls-5 | HIGH | Local code + official docs | `/Users/cjvana/Documents/GitHub/MeshCore/src/MeshCore.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.h`; https://docs.meshcore.io/companion_protocol/ |
| ITEM-pitfalls-6 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/security_utils.py` |
| ITEM-pitfalls-7 | HIGH | Local code + official docs | `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp`; https://docs.meshcore.io/companion_protocol/ |
| ITEM-pitfalls-8 | HIGH | Project requirement + local code | `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp` |
| ITEM-pitfalls-9 | HIGH | Local code + ecosystem docs/search | `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/SimpleMeshTables.h`; https://meshtastic.org/docs/overview/mesh-algo/ ; https://scholarworks.gnu.ac.kr/item/ccb29254-804b-413d-adc5-853fe697241b |
| ITEM-pitfalls-10 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/MeshCore/src/Packet.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/SimpleMeshTables.h` |
| ITEM-pitfalls-11 | HIGH | Local code/docs | `/Users/cjvana/Documents/GitHub/MeshCore/docs/payloads.md`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp` |
| ITEM-pitfalls-12 | HIGH | Local code + official docs | `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/main.cpp`; https://docs.meshcore.io/companion_protocol/ |
| ITEM-pitfalls-13 | HIGH | Local code + official docs | `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.h`; https://docs.meshcore.io/companion_protocol/ |
| ITEM-pitfalls-14 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/MeshCore/src/Mesh.h`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/message_handler.py` |
| ITEM-pitfalls-15 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/DataStore.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/NodePrefs.h` |
| ITEM-pitfalls-16 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/NodePrefs.h`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/DataStore.cpp` |
| ITEM-pitfalls-17 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/MeshCore/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/main.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/variants/heltec_v3/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/variants/rak4631/platformio.ini` |
| ITEM-pitfalls-18 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/MeshCore/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp` |

173
.forge/research/prior-art.md Normal file
View File

@@ -0,0 +1,173 @@
# Prior Art Research: Firmware-Resident MeshCore Bot
Checked: 2026-05-14
Scope: firmware-resident bots and embedded command responders in LoRa/mesh systems; existing MeshCore companion firmware, Python MeshCore bot, Colorado Mesh community bot/VPS coordinator patterns, and storage/flash/RAM feasibility for Heltec v3 and RAK4631 representative builds. This replaces stale coordinator-first conclusions: the recommended direction is a firmware-only bot with host-side projects used only as behavioral references.
### ITEM-prior-art-1: Upstream MeshCore companion firmware is the correct patch base
- **URL:** https://github.com/meshcore-dev/MeshCore
- **What it does well:** MeshCore provides the C++ mesh library and firmware examples for Companion, Repeater, Room Server, Sensor, and related roles. The local `examples/companion_radio` implementation already receives DMs and channel messages, queues offline messages, sends contact/channel text, exposes battery/storage reporting, stores contacts/channels/prefs, and has hooks (`onMessageRecv`, `onChannelMessageRecv`) exactly where a firmware bot can process commands before/alongside the serial companion interface. The latest fetched release was Companion Firmware v1.15.0, published 2026-04-19.
- **What it lacks:** No upstream firmware-resident command bot or multi-bot response election layer is visible. Companion firmware is still designed around an external app/host, so bot logic must be added without breaking phone/serial clients or changing the node role into a repeater.
- **What we can learn:** Use upstream MeshCore as a submodule and add Colorado Mesh firmware-bot code as a narrow companion-radio extension. Put command parsing in the receive hooks and response sending through existing MeshCore `sendMessage` / `sendGroupMessage` paths rather than inventing a parallel radio stack.
- **License:** MIT
- **Confidence:** HIGH
- **Source:** WebFetch + local code — https://github.com/meshcore-dev/MeshCore ; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`
- **Checked:** 2026-05-14
### ITEM-prior-art-2: MeshCore companion protocol supports the needed behavior but imposes short-message constraints
- **URL:** https://docs.meshcore.io/companion_protocol/
- **What it does well:** The companion protocol documents channel text send (`0x03`), channel binary datagrams (`0x3E`), queued-message polling (`0x0A`), async message-waiting notices, and battery/storage reporting (`0x14` response includes battery millivolts plus used/total storage KB). It states a text-message limit of 133 characters and a datagram payload limit of 163 bytes.
- **What it lacks:** It exposes used/total storage but no general-purpose arbitrary config/storage API for a bot. It is an app-to-firmware protocol, not an internal bot API, and BLE framing/MTU constraints make verbose responses inappropriate.
- **What we can learn:** Firmware bot responses must be terse and deterministic. Treat 133 characters as the safe response budget. Avoid large JSON/config payloads over chat. For bot configuration, prefer compile-time defaults plus a small firmware-side config structure saved with `DataStore`, not a full Python-style config file.
- **License:** Documentation / N/A
- **Confidence:** HIGH
- **Source:** WebFetch — https://docs.meshcore.io/companion_protocol/
- **Checked:** 2026-05-14
### ITEM-prior-art-3: agessaman meshcore-bot is the behavior oracle, not the runtime model
- **URL:** https://github.com/agessaman/meshcore-bot
- **What it does well:** The Python bot supports serial/BLE/TCP companion connections, configurable keywords, plugin commands, rate limiting, user bans, scheduled messages, DMs, logging, Discord/webhook integrations, weather/AQI/solar/sports/satellite feeds, stats, path diagnostics, repeater management, and web viewer features. Local command modules show a rich command set and a practical channel policy surface (`monitor_channels`, `respond_to_dms`).
- **What it lacks:** It assumes Python, asyncio, SQLite, HTTP/TLS clients, API keys, geocoding, dynamic plugins, and often internet reachability. Those features are unrealistic in first-pass firmware, especially on RAK4631-class nRF52840 targets.
- **What we can learn:** Port semantics in tiers. Firmware MVP should include: `ping`, `help/cmd`, `test`, `hello`, `dice`, `roll`, short `path/status` diagnostics from in-memory packet metadata, DM response support, #bot/#testing channel handling, #emergency-to-Public forwarding, per-command cooldown/rate limiting, passive listen-before-answer suppression, and known-bot trust. Defer: weather, AQI, sports, jokes via HTTP, satellite passes, solar forecast, Discord/web viewer, SQLite stats, dynamic plugins, repeater-management workflows, and feed parsing.
- **License:** MIT
- **Confidence:** HIGH
- **Source:** Local repo — `/Users/cjvana/Documents/GitHub/meshcore-bot/README.md`, `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/commands/*`, `/Users/cjvana/Documents/GitHub/meshcore-bot/requirements.txt`
- **Checked:** 2026-05-14
### ITEM-prior-art-4: Colorado Mesh community bot/VPS coordinator proves the desired coordination semantics
- **URL:** N/A
- **What it does well:** The local and read-only remote community bot patches host-side `send_response` / `send_channel_message` so DMs bypass coordination, channel responses are coordinated exactly once, and coordinator outages fall back to score-based delay. The community scoring model uses hop score, infrastructure/fan-in proxy, exact path familiarity, and freshness. The LAN host runtime config currently uses `monitor_channels = #bot,#emergency` and `emergency_broadcast_channel = Public`, matching the pivoted channel policy.
- **What it lacks:** The VPS coordinator requires IP reachability, registration/auth, PostgreSQL, Python containers, and host-side companion connectivity. It is the opposite of the off-grid firmware-only goal.
- **What we can learn:** Preserve the observable behavior, not the deployment. Firmware should key pending responses by a stable original-message hash, compute a compact local delivery score, schedule a delay, listen for a known bot's response to the same request, and cancel if another trusted bot answers first. DMs should remain immediate because only the addressed bot received them.
- **License:** Private / N/A
- **Confidence:** HIGH
- **Source:** Local repo + read-only SSH — `/Users/cjvana/Documents/GitHub/meshcore-community-bot/docs/COMMUNITY_DESIGN.md`, `/Users/cjvana/Documents/GitHub/meshcore-community-bot/community/message_interceptor.py`, `cj-vps:~/meshcore-community-bot`, `cjvana@10.0.0.222:~/meshcore-community-bot/config.ini`
- **Checked:** 2026-05-14
### ITEM-prior-art-5: Meshtastic firmware modules prove small embedded bot-like features are the norm
- **URL:** https://meshtastic.org/docs/configuration/module/canned-message/
- **What it does well:** Meshtastic ships firmware modules that generate mesh messages from device-side logic. Canned Message sends predefined messages without a phone and limits the combined preset list to 200 bytes. Detection Sensor monitors one GPIO and sends rate-limited mesh alerts. Remote Hardware performs addressed GPIO read/write/watch operations through firmware-controlled request/response behavior.
- **What it lacks:** These are narrow firmware modules, not a general text command bot. Remote Hardware is GPIO-specific and newer firmware may require custom builds; Canned Message needs an input peripheral; Detection Sensor is a one-pin alert module.
- **What we can learn:** The successful embedded pattern is deliberately small: fixed config, bounded text, simple state machines, rate limits, and compile-time/module flags. A MeshCore firmware bot should follow that model instead of embedding a general plugin runtime.
- **License:** Project/documentation / N/A
- **Confidence:** HIGH
- **Source:** WebFetch — https://meshtastic.org/docs/configuration/module/canned-message/ ; https://meshtastic.org/docs/configuration/module/detection-sensor/ ; https://meshtastic.org/docs/configuration/module/remote-hardware/
- **Checked:** 2026-05-14
### ITEM-prior-art-6: disaster.radio shows firmware-resident slash commands are practical on LoRa mesh devices
- **URL:** https://github.com/sudomesh/disaster-radio/blob/master/firmware/src/middleware/Console.cpp
- **What it does well:** The firmware console parses line-based slash commands such as `/help`, `/join`, `/nick`, `/raw`, `/lora`, `/get`, `/set`, and `/restart`. Commands update settings, broadcast info datagrams, restart the device, and use simple tokenization and bounded buffers.
- **What it lacks:** It is not MeshCore, is not a multi-bot responder election system, and has no direct compatibility with MeshCore companion channels or DMs.
- **What we can learn:** Keep the firmware command parser C-style and bounded: copy/terminate the input line, tokenize first word, dispatch to a fixed command table, validate all lengths, and use short replies. This is the right implementation style for `ping/help/test/dice/roll/status` in companion firmware.
- **License:** Repository license not verified here / N/A
- **Confidence:** HIGH
- **Source:** WebFetch — https://github.com/sudomesh/disaster-radio/blob/master/firmware/src/middleware/Console.cpp
- **Checked:** 2026-05-14
### ITEM-prior-art-7: LoRa_APRS_iGate proves embedded automation can be broad, but its heavy features are not MVP guidance
- **URL:** https://github.com/richonguzman/LoRa_APRS_iGate
- **What it does well:** This ESP32/LoRa APRS firmware performs iGate/digipeater automation, beacon scheduling, failover, packet filtering/blacklisting, telemetry, APRS queries, web configuration, OTA, MQTT, and sensor integrations. It targets many LoRa boards, including Heltec variants and RAK4631-class hardware.
- **What it lacks:** It is APRS-oriented, not MeshCore. Many features depend on WiFi/APRS-IS/MQTT/web subsystems and its GPL-3.0 license is not suitable for direct code reuse in an MIT MeshCore patch without a deliberate license strategy.
- **What we can learn:** Embedded LoRa firmware can do meaningful autonomous automation, but the MVP should not copy heavyweight web, MQTT, and internet-service layers. Copy the bounded scheduler/rate-limit/filtering concepts only.
- **License:** GPL-3.0
- **Confidence:** HIGH
- **Source:** WebFetch — https://github.com/richonguzman/LoRa_APRS_iGate
- **Checked:** 2026-05-14
### ITEM-prior-art-8: Cyclenerd meshcore-bot shows the safe minimal host-bot baseline
- **URL:** https://github.com/Cyclenerd/meshcore-bot
- **What it does well:** A small Node.js MeshCore bot over USB serial that responds only in private channels to avoid public-channel spam. It implements simple `.ping` and `.date` commands and can query/log repeater status.
- **What it lacks:** It is host-side, private-channel-only, and does not solve coordinated channel replies. It has no firmware component.
- **What we can learn:** If channel coordination is not ready, the safe fallback is DM-only operation plus explicit #bot/#testing opt-in. The firmware MVP can still be useful if it starts with DMs and restricted channels rather than Public.
- **License:** Apache-2.0
- **Confidence:** HIGH
- **Source:** WebFetch — https://github.com/Cyclenerd/meshcore-bot
- **Checked:** 2026-05-14
### ITEM-prior-art-9: MESH-API/MESH-AI host bots are useful behavior references but too heavy for firmware
- **URL:** https://github.com/mr-tbot/mesh-api
- **What it does well:** These host-side tools bridge Meshtastic/MeshCore, MQTT, Discord, AI/API providers, custom commands, emergency commands, origin tags, duplicate buffers, and randomized command aliases. They explicitly account for loop prevention and multi-system routing.
- **What it lacks:** They require a host runtime, network services, API integrations, and GPL-3.0 licensing. They are not firmware-resident bots or MeshCore companion patches.
- **What we can learn:** Adopt origin markers and duplicate buffers conceptually. Do not port AI/API routes into firmware. A useful firmware response format could include a compact bot marker and original-message hash suffix when needed for suppression/debugging.
- **License:** GPL-3.0 for mesh-api as fetched
- **Confidence:** MEDIUM
- **Source:** WebSearch/WebFetch — https://github.com/mr-tbot/mesh-api ; https://github.com/mr-tbot/mesh-ai
- **Checked:** 2026-05-14
### ITEM-prior-art-10: Existing MeshCore storage model has enough persistent space for small bot config, not host-bot databases
- **URL:** https://github.com/meshcore-dev/MeshCore
- **What it does well:** `DataStore` already persists identity, node prefs, contacts, group channels, and advert blobs; it reports used/total storage in KB. On nRF52 with `EXTRAFS`, companion firmware creates `CustomLFS ExtraFS(0xD4000, 0x19000, 128)`, i.e. about 100 KB of secondary LittleFS-style storage. Contacts are fixed records of about 153 bytes each, channels about 68 bytes each, and advert blobs are fixed bounded records. ESP32 uses SPIFFS and reports `SPIFFS.usedBytes()` / `SPIFFS.totalBytes()`.
- **What it lacks:** There is no existing firmware database analogous to Python SQLite stats, no large config store, and storage competes with contacts/channels/adverts. RAK4631-class nRF52 storage is especially finite.
- **What we can learn:** Store only tiny bot state persistently: enable flag, allowed channel names/indices, a few known-bot public-key prefixes, cooldown settings, and maybe a dozen canned response strings. Keep history/suppression volatile. Do not port Python stats, feed caches, or repeater analytics DB into firmware.
- **License:** MIT
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/DataStore.cpp`, `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp`, `/Users/cjvana/Documents/GitHub/MeshCore/variants/rak4631/platformio.ini`
- **Checked:** 2026-05-14
### ITEM-prior-art-11: Representative device space supports a firmware bot MVP, with RAK4631 as the limiting target
- **URL:** https://docs.rakwireless.com/product-categories/wisblock/rak4631/overview/
- **What it does well:** Heltec WiFi LoRa 32 V3 uses an ESP32-S3N8 with 8 MB integrated flash and SX1262 radio. RAK4631 uses Nordic nRF52840 with 1 MB flash, 256 KB RAM, 64 MHz Cortex-M4, and SX1262. MeshCore `platformio.ini` caps RAK4631 companion uploads at 712,704 bytes. Release asset sizes for MeshCore Companion v1.15.0 were: Heltec v3 USB non-merged 629,424 bytes, Heltec v3 BLE non-merged 1,262,752 bytes, RAK4631 USB UF2 955,392 bytes / ZIP 478,359 bytes, and RAK4631 BLE UF2 971,264 bytes / ZIP 486,343 bytes. UF2/ZIP sizes are packaging sizes, not exact linked ELF flash usage, so exact headroom requires a local build.
- **What it lacks:** I did not run fresh PlatformIO builds or measure patched `.elf` sections, so exact free flash/RAM after adding bot code is still missing. RAK4631 headroom cannot be safely inferred from compressed ZIP alone.
- **What we can learn:** Design to the RAK4631 cap first. A realistic firmware-bot MVP should target roughly 15-40 KB additional flash, 2-8 KB additional RAM, and under 2 KB persistent config. That budget fits simple parsers, fixed command tables, response strings, pending-response suppression, known-bot prefixes, and channel policy. It does not fit TLS/HTTP clients, geocoders, ephemeris libraries, SQLite-style history, web UIs, or dynamic plugins. Heltec v3 has much more flash margin; RAK4631 should be the release gate.
- **License:** Hardware docs / N/A
- **Confidence:** MEDIUM
- **Source:** WebFetch + gh release + local code — https://docs.heltec.org/en/node/esp32/wifi_lora_32/index.html ; https://docs.rakwireless.com/product-categories/wisblock/rak4631/overview/ ; https://www.nordicsemi.com/products/nrf52840 ; `gh release view companion-v1.15.0 --repo meshcore-dev/MeshCore`; `/Users/cjvana/Documents/GitHub/MeshCore/variants/rak4631/platformio.ini`
- **Checked:** 2026-05-14
### ITEM-prior-art-12: Public-channel emergency handling should be explicit, not an accidental bridge of all bot traffic
- **URL:** N/A
- **What it does well:** The existing local/remote Python/community bot configs show normal monitored channels as #bot/#emergency and an emergency broadcast target of Public. The pivoted project policy refines this: normal bot traffic belongs in private DMs, #bot, and #testing; #emergency should be routed/announced to Public with `EMERGENCY MESSAGE FROM <user>` plus the original text.
- **What it lacks:** Current host-side code also contains Discord webhook forwarding and richer emergency formatting. Firmware cannot assume internet, Discord, or webhook reachability.
- **What we can learn:** Implement #emergency as a special firmware path: do not answer with normal bot help/noise; forward a concise Public alert using the configured node/user identity and original text, then rate-limit to avoid loops. Treat Public as emergency output only for MVP.
- **License:** Private config / N/A
- **Confidence:** HIGH
- **Source:** Local + read-only SSH + project brief — `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`, `cjvana@10.0.0.222:~/meshcore-community-bot/config.ini`, `/Users/cjvana/Documents/GitHub/meshcore-community-bot/community/message_interceptor.py`
- **Checked:** 2026-05-14
## Firmware Feature Portability Assessment
| Python/community bot feature | Firmware MVP status | Reason |
|---|---:|---|
| DM command handling | First | Already present receive path; no multi-bot election needed. |
| #bot/#testing command handling | First | Small channel gate plus passive suppression. |
| #emergency to Public forwarding | First | Project requirement; simple string transform and strict rate limit. |
| `ping`, `help`, `cmd`, `test`, `hello` | First | Fixed strings and small parser. |
| `dice`, `roll` | First | Tiny random-number commands using existing RNG. |
| `path` / basic heard status | First/second | Use packet path/SNR metadata and existing advert path table; keep output short. |
| Passive listen-before-answer suppression | First | Core reason for firmware bot; bounded pending table. |
| Known bot identity trust | First | Small prefix list or full-key list; avoids suppressing on spoofed user text. |
| `stats` | Later, tiny version only | Volatile counters are feasible; SQLite-style history is not. |
| `repeater` management | Later | Contact-store operations are delicate and UI-heavy. |
| Weather/AQI/sports/jokes/dadjoke/feeds | Too heavy for firmware MVP | HTTP/TLS, parsing, API keys, caching, and internet dependency. |
| Satellite/solar forecast/geocoding | Too heavy for firmware MVP | Large math/data/API dependencies. |
| Discord/web viewer/MQTT/analytics | Too heavy/off-device | Requires IP services and host/web stack. |
| Dynamic plugins/i18n translation files | Too heavy | Firmware should use compile-time command table and fixed strings. |
## Confidence Summary
| Item ID | Level | Source Type | URL/Reference |
|---------|-------|-------------|---------------|
| ITEM-prior-art-1 | HIGH | WebFetch + local code | https://github.com/meshcore-dev/MeshCore ; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.cpp` |
| ITEM-prior-art-2 | HIGH | WebFetch | https://docs.meshcore.io/companion_protocol/ |
| ITEM-prior-art-3 | HIGH | Local repo | `/Users/cjvana/Documents/GitHub/meshcore-bot/README.md`, command modules, requirements |
| ITEM-prior-art-4 | HIGH | Local repo + read-only SSH | community bot design/interceptor and runtime configs |
| ITEM-prior-art-5 | HIGH | WebFetch | https://meshtastic.org/docs/configuration/module/canned-message/ ; https://meshtastic.org/docs/configuration/module/detection-sensor/ ; https://meshtastic.org/docs/configuration/module/remote-hardware/ |
| ITEM-prior-art-6 | HIGH | WebFetch | https://github.com/sudomesh/disaster-radio/blob/master/firmware/src/middleware/Console.cpp |
| ITEM-prior-art-7 | HIGH | WebFetch | https://github.com/richonguzman/LoRa_APRS_iGate |
| ITEM-prior-art-8 | HIGH | WebFetch | https://github.com/Cyclenerd/meshcore-bot |
| ITEM-prior-art-9 | MEDIUM | WebSearch/WebFetch | https://github.com/mr-tbot/mesh-api ; https://github.com/mr-tbot/mesh-ai |
| ITEM-prior-art-10 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/DataStore.cpp` |
| ITEM-prior-art-11 | MEDIUM | WebFetch + gh release + local code | Heltec docs, RAK docs, Nordic docs, MeshCore v1.15.0 release assets, RAK4631 platformio cap |
| ITEM-prior-art-12 | HIGH | Local config + project brief | `.forge/PROJECT.md`, LAN community-bot config, community message interceptor |

148
.forge/research/stack.md Normal file
View File

@@ -0,0 +1,148 @@
# Stack Research: Firmware-only MeshCore Bot
Checked: 2026-05-14
### ITEM-stack-1: Build in upstream MeshCore's Arduino C++/PlatformIO firmware stack
- **Recommendation:** Build the firmware-only bot in C++ inside the MeshCore companion firmware stack, using Arduino framework and PlatformIO exactly as upstream does. Keep `meshcore-bot` as behavioral reference only.
- **Rationale:** Current MeshCore describes itself as a compact C++ embedded LoRa/packet-radio library, and current upstream `platformio.ini` uses `framework = arduino` with architecture bases for ESP32, nRF52, RP2040, and STM32. The Python `meshcore-bot` connects over serial/BLE/TCP and implements a plugin bot outside the radio; it is useful for command semantics, but not for reducing firmware advert/response behavior on-device.
- **Confidence:** HIGH
- **Source:** Official docs + local code — https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/README.md; https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/platformio.ini; `/Users/cjvana/Documents/GitHub/meshcore-bot/README.md`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not introduce ESP-IDF-only, Zephyr, Rust, embedded Python/JS, Docker, or a VPS-side coordinator as the product stack; they either break MeshCore's cross-device firmware model or preserve the external-bot architecture being replaced.
### ITEM-stack-2: Keep MeshCore as an upstream submodule plus Colorado overlay/patch queue
- **Recommendation:** Structure the Colorado Mesh repo as a wrapper with `upstream/MeshCore` as a pinned git submodule, a `colorado/` overlay for new source files/config snippets/tests, and a deterministic `patches/` or `git-format-patch` queue applied to the submodule during build. Do not keep local edits directly inside the submodule as the only source of truth.
- **Rationale:** The user has selected submodule/patch-base strategy. MeshCore's PlatformIO project expects builds from the MeshCore repo root (`extra_configs = variants/*/platformio.ini`, relative `build_src_filter`, `build.sh`, and variant paths), so the wrapper should stage/apply patches into a worktree and invoke upstream `build.sh` from there. This preserves an auditable upstream pin while still allowing changes across `examples/companion_radio`, `src/helpers`, `docs`, and selected variant `.ini` files.
- **Confidence:** HIGH
- **Source:** Local code + project decision — `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/build.sh`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not vendor-copy MeshCore into the Colorado repo; upstream is moving. Do not rely on uncommitted submodule edits; they make rebuilds and upstream syncs non-reproducible. Do not place a separate PlatformIO project above MeshCore unless it merely orchestrates the upstream-root build.
### ITEM-stack-3: Use current MeshCore main as the baseline, then pin it
- **Recommendation:** Initialize the submodule from current `meshcore-dev/MeshCore` main, record the exact commit SHA in the Colorado repo, and refresh intentionally. Validate against the local checkout only as read-only reconnaissance.
- **Rationale:** The local MeshCore checkout is usable for inspection, but current upstream has evolved: current upstream `platformio.ini` has LoRa defaults `LORA_FREQ=869.618`, `LORA_BW=62.5`, `LORA_SF=8`, adds `-D ESP32_PLATFORM`, and nRF52 extra scripts include `create-uf2.py` and `patch_bluefruit.py`, while the local checkout differs. Starting from current upstream reduces protocol/build drift.
- **Confidence:** MEDIUM
- **Source:** Official source + local git — https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/platformio.ini; `/Users/cjvana/Documents/GitHub/MeshCore` commit `6b52fb32`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not base implementation solely on the stale local tree; it may miss current companion protocol and build changes. Do not float the submodule without a pinned commit; firmware builds need reproducibility.
### ITEM-stack-4: Preserve upstream dependency/platform pins for the first bot build
- **Recommendation:** Keep upstream platform and library pins initially: `platformio/espressif32@6.11.0` for ESP32 Arduino builds, `nordicnrf52` with `framework-arduinoadafruitnrf52 @ 1.10700.0` for nRF52, RadioLib `^7.3.0`, Crypto `^0.4.0`, RTClib `^2.1.3`, Melopero RV3028 `^1.1.0`, CayenneLPP `1.6.1`, and `densaugeo/base64 @ ~1.4.0` in companion targets.
- **Rationale:** The first risk is firmware behavior, not dependency modernization. `platformio/espressif32@6.11.0` with Arduino uses Arduino-ESP32 2.0.17, and RadioLib 7.3.0 is a 2025 release although newer RadioLib releases exist. Upgrading radio/platform dependencies while adding bot logic would confound failures across ESP32 and nRF52.
- **Confidence:** HIGH
- **Source:** Official docs/search + upstream source — https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/platformio.ini; https://github.com/platformio/platform-espressif32/blob/v6.11.0/platform.json; https://github.com/jgromes/RadioLib/releases/tag/7.3.0; https://docs.platformio.org/en/latest/librarymanager/dependencies.html
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not upgrade to Arduino-ESP32 3.x, latest RadioLib, or custom nRF framework packages until baseline bot patches build and run; dependency churn should be a separate phase.
### ITEM-stack-5: Add bot logic as small fixed-storage companion helpers
- **Recommendation:** Implement the bot as small C++ helper classes called by `examples/companion_radio/MyMesh`, with fixed-size arrays/ring buffers and compile-time feature flags such as `CMESH_BOT_ENABLED`, `CMESH_BOT_MAX_PENDING`, and `CMESH_BOT_MAX_KNOWN_BOTS`.
- **Rationale:** Companion `MyMesh` already owns the required hooks: private DM receive (`onMessageRecv`), group channel receive (`onChannelMessageRecv`), control/raw receive, channel lookup, contact lookup, and group/direct send (`sendGroupMessage`, `sendMessage`). Upstream contribution guidance says to keep embedded code concise and avoid dynamic allocation except during setup/begin. A helper object avoids a large framework while keeping Colorado logic reviewable.
- **Confidence:** HIGH
- **Source:** Local code + upstream README — `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.h`; https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/README.md
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not port the Python plugin loader, SQLite-backed stats, API clients, or dynamic command registry to firmware. Do not create a general embedded scripting runtime.
### ITEM-stack-6: Handle only firmware-feasible bot behavior in phase 1
- **Recommendation:** Phase 1 should implement deterministic local commands and routing policy: `ping`, `test`, compact `help`, `cmd`, simple `dice/roll`, path/channel echo where already available, passive suppression/listen-before-answer, #bot/#testing routing, private DM replies, and #emergency forwarding to Public. Defer weather, AQI, satellite, sports, jokes from web APIs, repeater database management, and web viewer features.
- **Rationale:** `meshcore-bot` includes many network/API/database features that assume Python, internet, filesystem logs, and async plugins. Firmware bot value is local decentralized operation and duplicate suppression. Keeping responses short also fits MeshCore `MAX_TEXT_LEN = 10*CIPHER_BLOCK_SIZE = 160` bytes and group messages include a sender-name prefix.
- **Confidence:** HIGH
- **Source:** Local code/docs — `/Users/cjvana/Documents/GitHub/meshcore-bot/README.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.h`; `/Users/cjvana/Documents/GitHub/MeshCore/docs/payloads.md`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not attempt feature parity with host-side `meshcore-bot`; API-backed commands are not off-grid and would expand flash/RAM/storage unnecessarily.
### ITEM-stack-7: Use MeshCore group/private message APIs for bot traffic, not new transport timing knobs
- **Recommendation:** Send normal bot replies through existing `sendMessage`/`sendGroupMessage` paths and implement bot-specific delay/suppression above the radio layer. Use configured channel names/hash lookup to restrict normal traffic to private DMs, #bot, and #testing, and special-case #emergency to publish an `EMERGENCY MESSAGE FROM <user>` announcement to Public.
- **Rationale:** MeshCore already parses private text and group text into `onMessageRecv`/`onChannelMessageRecv`, and packet docs define group text as encrypted channel payload with `<sender name>: <message body>`. The project decision explicitly says not to repurpose lower-layer MeshCore `txdelay`, `direct.txdelay`, or `rxdelay` as bot election controls.
- **Confidence:** HIGH
- **Source:** Local code/docs + project decision — `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/docs/payloads.md`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not route normal bot chatter on Public. Do not use lower-layer TX/RX timing knobs as the primary coordinator; it risks affecting mesh behavior beyond bot replies.
### ITEM-stack-8: Build representative firmware with upstream build.sh before expanding
- **Recommendation:** Use upstream `build.sh` from inside the patched MeshCore worktree. During development, build only `Heltec_v3_companion_radio_usb`, `Heltec_v3_companion_radio_ble`, `RAK_4631_companion_radio_usb`, and `RAK_4631_companion_radio_ble`; after those pass, run `build.sh build-companion-firmwares` in CI.
- **Rationale:** Upstream `build.sh build-firmware <env>` injects firmware version/build date, runs `pio run -e`, creates merged ESP32 `.bin` images, creates nRF52 `.uf2`, and writes outputs to `out/`. The representative set covers ESP32-S3 and nRF52840 plus USB/BLE companion variants without paying the cost of every supported board during inner-loop development.
- **Confidence:** HIGH
- **Source:** Local CI/build scripts — `/Users/cjvana/Documents/GitHub/MeshCore/build.sh`; `/Users/cjvana/Documents/GitHub/MeshCore/.github/workflows/build-companion-firmwares.yml`; `/Users/cjvana/Documents/GitHub/MeshCore/.github/actions/setup-build-environment/action.yml`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not hand-maintain an independent board matrix first; upstream environment suffixes and artifact logic already encode release behavior. Do not wait for all companion boards before validating the two requested representatives.
### ITEM-stack-9: Heltec v3 companion target is comfortably feasible
- **Recommendation:** Treat Heltec v3 as the easiest first target. Use current upstream `Heltec_v3_companion_radio_usb` and `Heltec_v3_companion_radio_ble` envs unchanged except for the bot feature flag/overlay patch.
- **Rationale:** Current upstream Heltec v3 extends `Heltec_lora32_v3`, uses board `esp32-s3-devkitc-1`, ESP32-S3 Arduino, SSD1306 UI, `MAX_CONTACTS=350`, `MAX_GROUP_CHANNELS=40`, and BLE adds `OFFLINE_QUEUE_SIZE=256`. PlatformIO's `esp32-s3-devkitc-1` board uses 8 MB flash with `default_8MB.csv`; app0/app1 are 0x330000 each (3,342,336 bytes), SPIFFS is 0x180000 (1,572,864 bytes). Heltec official specs confirm ESP32-S3FN8/ESP32-S3N8 with 8 MB flash and no PSRAM.
- **Confidence:** HIGH
- **Source:** Official source/docs — https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/variants/heltec_v3/platformio.ini; https://raw.githubusercontent.com/platformio/platform-espressif32/master/boards/esp32-s3-devkitc-1.json; https://raw.githubusercontent.com/espressif/arduino-esp32/master/tools/partitions/default_8MB.csv; https://docs.heltec.cn/en/node/esp32/wifi_lora_32/index.html; https://resource.heltec.cn/download/WiFi_LoRa_32_V3/HTIT-WB32LA_V3.2.pdf
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not optimize for Heltec v3 storage first; its app and filesystem margins are much larger than RAK4631's. Do not depend on PSRAM; Heltec v3 does not provide it.
### ITEM-stack-10: RAK4631 is feasible but is the storage/RAM gatekeeper
- **Recommendation:** Treat RAK4631 BLE companion as the gating target. Keep the firmware bot under roughly 20 KB additional flash and 2 KB additional static RAM until measured builds prove more margin.
- **Rationale:** Current upstream RAK4631 companion USB/BLE extend `rak4631`, use `boards/nrf52840_s140_v6_extrafs.ld`, and explicitly set `board_upload.maximum_size = 712704`. The extra-FS linker gives an app FLASH region of 712,704 bytes and RAM region of 237,568 bytes. RAK's raw nRF52840 has 1 MB flash/256 KB RAM, but SoftDevice, bootloader/settings, and extra filesystem reserve much of that; the companion app region, not raw flash, is the practical limit.
- **Confidence:** HIGH
- **Source:** Official source/docs — https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/variants/rak4631/platformio.ini; https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/boards/rak4631.json; https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/boards/nrf52840_s140_v6_extrafs.ld; https://docs.rakwireless.com/product-categories/wisblock/rak4631/overview/; https://www.nordicsemi.com/Products/nRF52840
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not assume the full 1 MB nRF52840 flash is available to the app. Do not store large response tables or logs in RAK4631 internal flash without measuring filesystem pressure.
### ITEM-stack-11: Firmware bot storage budget is small enough if responses stay static and ephemeral
- **Recommendation:** Budget the phase-1 firmware bot at approximately 10-25 KB flash, 0.5-2 KB RAM, and 0-4 KB persistent storage. Require CI to report binary size deltas for Heltec v3 and RAK4631 on every PR.
- **Rationale:** A minimal command parser, static command table, short static response strings, a small known-bot/suppression table, and 4-8 pending response records fit easily: example estimate is 3-8 KB `.text`, 2-10 KB `.rodata` depending on help text, 512-1536 B `.bss` for suppression state, and <512 B stack per invocation if implemented without large local buffers. MeshCore already carries large state for companion mode: approximate struct sizing from source gives `ContactInfo` ~184 B, `ChannelDetails` ~56 B, offline frame ~173 B, and `Packet` ~260 B; existing `MAX_CONTACTS=350`, `MAX_GROUP_CHANNELS=40`, and BLE `OFFLINE_QUEUE_SIZE=256` dominate RAM more than the bot should. This estimate could not be verified by compiling because `pio` is not installed in the local environment.
- **Confidence:** MEDIUM
- **Source:** Local source analysis — `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/ContactInfo.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/ChannelDetails.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/Packet.h`; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.h`; local command result: `pio: command not found`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not embed full help prose, translation files, jokes, channel databases, repeater databases, web/API clients, or persistent usage stats in firmware phase 1; those could turn a <25 KB feature into a storage problem.
### ITEM-stack-12: Storage feasibility estimate by representative target
- **Recommendation:** Proceed with firmware-only bot if the first implementation stays within the conservative budget below; block or trim features if measured RAK4631 BLE app flash delta exceeds 25 KB or static RAM delta exceeds 2 KB.
- **Rationale:** Heltec v3 has about 3.34 MB per OTA app slot and 1.57 MB SPIFFS under the default 8 MB partition table, so a 10-25 KB bot is negligible (<0.8% of one app slot). RAK4631 companion has a 712,704-byte app region and 237,568-byte RAM region; the same bot is ~1.4-3.5% of the app region and ~0.2-0.9% of RAM. Persistent config should be a few hundred bytes; even a 4 KB settings/log reserve is reasonable but should be optional. The unknown is current baseline binary size, not the expected bot delta, because no local PlatformIO build artifacts exist.
- **Confidence:** MEDIUM
- **Source:** Official docs + local analysis — https://raw.githubusercontent.com/espressif/arduino-esp32/master/tools/partitions/default_8MB.csv; https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/boards/nrf52840_s140_v6_extrafs.ld; https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/variants/rak4631/platformio.ini; `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.h`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not declare final binary margin until PlatformIO builds run; use this as feasibility guidance, then replace with measured `.text/.data/.bss` and firmware size deltas.
### ITEM-stack-13: Add a size-reporting CI step before feature growth
- **Recommendation:** Add a wrapper command such as `scripts/build-representative.sh` that applies patches, runs the four representative envs, captures PlatformIO RAM/flash usage output, records artifact byte sizes, and fails on configurable bot-delta thresholds once a baseline is established.
- **Rationale:** The user's explicit question is storage feasibility. PlatformIO normally reports memory usage per environment, but local `pio` is unavailable and there are no existing `.pio` artifacts. CI size reports make the feasibility estimate concrete and keep future feature creep visible, especially on RAK4631 BLE.
- **Confidence:** HIGH
- **Source:** Local build scripts + official PlatformIO behavior — `/Users/cjvana/Documents/GitHub/MeshCore/build.sh`; https://docs.platformio.org/en/latest/core/userguide/cmd_run.html
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not rely on source-only estimates after code exists. Do not run only Heltec v3; RAK4631 is the constrained representative.
### ITEM-stack-14: Use host-side Python only for tests and behavioral fixtures
- **Recommendation:** Keep Python in the repo only for development tooling: generating patch worktrees, golden command-response fixtures, fake MeshCore frames, and CI parsing of size reports. Firmware behavior should not depend on Python at runtime.
- **Rationale:** `meshcore-bot` is rich behavioral prior art: command keywords, rate limits, channel policy, and response wording. But its async plugin loader, HTTP API clients, SQLite/database managers, web viewer, and serial/BLE/TCP connection code cannot run inside companion firmware. Host-side tests can still prevent regressions when translating selected behavior to C++.
- **Confidence:** HIGH
- **Source:** Local code — `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/commands`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/command_manager.py`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/message_handler.py`
- **Checked:** 2026-05-14
- **Alternatives rejected:** Do not make the firmware call out to a local Python service for normal bot responses; that recreates the coordinator dependency.
## Confidence Summary
| Item ID | Level | Source Type | URL/Reference |
|---------|-------|-------------|---------------|
| ITEM-stack-1 | HIGH | Official docs + Local code | https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/README.md; https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/platformio.ini; `/Users/cjvana/Documents/GitHub/meshcore-bot/README.md` |
| ITEM-stack-2 | HIGH | Local code + Project decision | `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/platformio.ini`; `/Users/cjvana/Documents/GitHub/MeshCore/build.sh` |
| ITEM-stack-3 | MEDIUM | Official source + Local git | https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/platformio.ini; `/Users/cjvana/Documents/GitHub/MeshCore` |
| ITEM-stack-4 | HIGH | Official docs/search + Upstream source | https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/platformio.ini; https://github.com/platformio/platform-espressif32/blob/v6.11.0/platform.json; https://github.com/jgromes/RadioLib/releases/tag/7.3.0; https://docs.platformio.org/en/latest/librarymanager/dependencies.html |
| ITEM-stack-5 | HIGH | Local code + Upstream README | `/Users/cjvana/Documents/GitHub/MeshCore/examples/companion_radio/MyMesh.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.h`; https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/README.md |
| ITEM-stack-6 | HIGH | Local code/docs | `/Users/cjvana/Documents/GitHub/meshcore-bot/README.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.h`; `/Users/cjvana/Documents/GitHub/MeshCore/docs/payloads.md` |
| ITEM-stack-7 | HIGH | Local code/docs + Project decision | `/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.forge/PROJECT.md`; `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/BaseChatMesh.cpp`; `/Users/cjvana/Documents/GitHub/MeshCore/docs/payloads.md` |
| ITEM-stack-8 | HIGH | Local CI/build scripts | `/Users/cjvana/Documents/GitHub/MeshCore/build.sh`; `/Users/cjvana/Documents/GitHub/MeshCore/.github/workflows/build-companion-firmwares.yml` |
| ITEM-stack-9 | HIGH | Official source/docs | https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/variants/heltec_v3/platformio.ini; https://raw.githubusercontent.com/platformio/platform-espressif32/master/boards/esp32-s3-devkitc-1.json; https://raw.githubusercontent.com/espressif/arduino-esp32/master/tools/partitions/default_8MB.csv; https://docs.heltec.cn/en/node/esp32/wifi_lora_32/index.html |
| ITEM-stack-10 | HIGH | Official source/docs | https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/variants/rak4631/platformio.ini; https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/boards/rak4631.json; https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/boards/nrf52840_s140_v6_extrafs.ld; https://docs.rakwireless.com/product-categories/wisblock/rak4631/overview/ |
| ITEM-stack-11 | MEDIUM | Local source analysis | `/Users/cjvana/Documents/GitHub/MeshCore/src/helpers/ContactInfo.h`; `/Users/cjvana/Documents/GitHub/MeshCore/src/Packet.h`; local `pio` availability check |
| ITEM-stack-12 | MEDIUM | Official docs + Local analysis | https://raw.githubusercontent.com/espressif/arduino-esp32/master/tools/partitions/default_8MB.csv; https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/boards/nrf52840_s140_v6_extrafs.ld; https://raw.githubusercontent.com/meshcore-dev/MeshCore/main/variants/rak4631/platformio.ini |
| ITEM-stack-13 | HIGH | Local build scripts + Official docs | `/Users/cjvana/Documents/GitHub/MeshCore/build.sh`; https://docs.platformio.org/en/latest/core/userguide/cmd_run.html |
| ITEM-stack-14 | HIGH | Local code | `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/commands`; `/Users/cjvana/Documents/GitHub/meshcore-bot/modules/command_manager.py` |

47
.forge/reviews/claude-step-1.json Normal file
View File

@@ -0,0 +1,47 @@
{
"verdict": "APPROVED",
"blocking_issues": 0,
"summary": "Step 1 staged changes align with the master plan and step execution plan. Prior blockers are resolved: dirty checks include untracked submodule files, and repo-root detection uses a subshell for the final cwd change.",
"reviewed_files": [
"/Users/cjvana/Documents/GitHub/meshcore-bot-fw/.gitmodules",
"/Users/cjvana/Documents/GitHub/meshcore-bot-fw/README.md",
"/Users/cjvana/Documents/GitHub/meshcore-bot-fw/colorado/README.md",
"/Users/cjvana/Documents/GitHub/meshcore-bot-fw/patches/meshcore/.gitkeep",
"/Users/cjvana/Documents/GitHub/meshcore-bot-fw/scripts/apply-patches.sh",
"/Users/cjvana/Documents/GitHub/meshcore-bot-fw/scripts/export-patches.sh",
"/Users/cjvana/Documents/GitHub/meshcore-bot-fw/scripts/meshcore-env.sh",
"/Users/cjvana/Documents/GitHub/meshcore-bot-fw/vendor/MeshCore"
],
"diff_basis": "provided_list",
"plan_section": "Step 1: Bootstrap wrapper repository, upstream submodule, and patch workflow",
"dimensions": {
"plan_alignment": {
"score": "PASS",
"notes": "Implements the planned wrapper repository shape, pinned MeshCore submodule, empty patch queue, shared environment helper, patch apply script, export script, and README notes."
},
"correctness_safety": {
"score": "PASS",
"notes": "Submodule existence and dirty-state checks are present; dirty checks use git status --porcelain and therefore include untracked files. Empty patch queue exits successfully without modifying firmware files."
},
"code_quality": {
"score": "PASS",
"notes": "Scripts are simple, readable, use strict shell settings where executable workflow scripts need them, and quote path variables correctly."
},
"completeness": {
"score": "PASS",
"notes": "Acceptance criteria are satisfied based on reviewed code and provided verification. Representative env names match the plan."
},
"patterns": {
"score": "PASS",
"notes": "The implementation follows the wrapper-plus-submodule-plus-deterministic-patch-queue architecture required for later Forge steps."
},
"integration": {
"score": "PASS",
"notes": "Submodule metadata points at the expected upstream URL and staged submodule pointer is pinned at 910b1bee5b0ccffc472c7684d4165fc85c681896."
}
},
"required_changes": [],
"flags": [],
"suggestions": [],
"confidence": "high"
}

56
.forge/steps/step-1-plan.md Normal file
View File

@@ -0,0 +1,56 @@
# Step 1 Execution Plan: Bootstrap wrapper repository, upstream submodule, and patch workflow
## Goal
Create a reproducible Colorado Mesh firmware wrapper that pins upstream MeshCore and can apply/export Colorado bot patches without relying on untracked submodule edits.
## Current Code Observations
- Local MeshCore exists at `/Users/cjvana/Documents/GitHub/MeshCore` commit `6b52fb32301c273fc78d96183501eb23ad33c5bb`, but research recommends pinning the wrapper submodule independently.
- MeshCore builds from its repo root with `build.sh build-firmware <env>` and expects `FIRMWARE_VERSION` in the environment before building.
- MeshCore `platformio.ini` includes all variant configs via `variants/*/platformio.ini` and currently enables `ENABLE_PRIVATE_KEY_IMPORT=1` / `ENABLE_PRIVATE_KEY_EXPORT=1`; key hardening is a later step.
- Representative environments exist in the local checkout: `Heltec_v3_companion_radio_usb`, `Heltec_v3_companion_radio_ble`, `RAK_4631_companion_radio_usb`, and `RAK_4631_companion_radio_ble`.
- PlatformIO is not installed locally right now, so Step 1 verification must be limited to git/submodule/script behavior and shell syntax.
## Files to Change
- `.gitmodules` — created by `git submodule add`.
- `vendor/MeshCore` — submodule pointer to upstream MeshCore.
- `scripts/meshcore-env.sh` — shared wrapper environment variables and helper functions.
- `scripts/apply-patches.sh` — verify submodule state and apply sorted patch queue.
- `scripts/export-patches.sh` — regenerate patch queue from a named submodule branch/range.
- `patches/meshcore/.gitkeep` — keep empty patch queue directory tracked.
- `colorado/README.md` — short notes for Colorado overlay intent.
- `README.md` — wrapper workflow instructions for future contributors.
## Ordered Implementation Checklist
1. Add `vendor/MeshCore` as a git submodule from `https://github.com/meshcore-dev/MeshCore.git` and leave it pinned at the resolved commit.
2. Create `scripts/`, `patches/meshcore/`, and `colorado/` directories.
3. Implement `scripts/meshcore-env.sh` with repo-root detection, `MESHCORE_DIR`, `PATCH_DIR`, representative env list, and `meshcore_commit()` helper.
4. Implement `scripts/apply-patches.sh` with strict shell settings, submodule existence check, dirty submodule check, empty-patch success path, sorted patch application, and clear status output.
5. Implement `scripts/export-patches.sh` with strict shell settings, dirty submodule guidance, configurable base ref defaulting to `origin/main`, deterministic patch regeneration, and refusal to export when there are no commits beyond the base.
6. Add minimal README/development notes describing submodule init, patch apply, patch export, and representative env names.
7. Run shell syntax checks and `scripts/apply-patches.sh` with the empty patch queue.
8. Stage only Step 1 files for review.
## Interfaces and Data Contracts
- `scripts/meshcore-env.sh` can be sourced by other scripts and must not execute builds itself.
- `scripts/apply-patches.sh` exits 0 with an empty patch queue and exits non-zero if the submodule is missing, dirty, or patch application fails.
- `scripts/export-patches.sh [base-ref]` regenerates `patches/meshcore/*.patch` from commits after `base-ref`; it does not export uncommitted submodule edits.
- Representative env names are exposed as the shell array `REPRESENTATIVE_ENVS`.
## Verification Plan
- Automated:
- `git submodule status`
- `bash -n scripts/meshcore-env.sh scripts/apply-patches.sh scripts/export-patches.sh`
- `bash scripts/apply-patches.sh`
- `git status --short`
- Manual:
- Confirm `vendor/MeshCore` points to a real upstream commit.
- Confirm an empty patch queue leaves the submodule unchanged.
- Regression:
- No firmware source files should be modified in Step 1.
- `.forge/` artifacts are not staged as product source except required Forge documents remain available for review.
## Stop Conditions
- Stop and ask if `git submodule add` fails due to network/auth issues.
- Stop and ask if `vendor/MeshCore` already exists with unexpected contents.
- Stop and ask before deleting or overwriting any existing non-Forge files.
- Stop if the submodule checkout has unexpected dirty changes immediately after adding it.

3
.gitmodules vendored Normal file
View File

@@ -0,0 +1,3 @@
[submodule "vendor/MeshCore"]
path = vendor/MeshCore
url = https://github.com/meshcore-dev/MeshCore.git

32
README.md Normal file
View File

@@ -0,0 +1,32 @@
# Colorado MeshCore Bot Firmware
Wrapper repository for Colorado Mesh firmware-only bot work on top of upstream MeshCore companion firmware.
## Layout
- `vendor/MeshCore/` — pinned upstream MeshCore submodule.
- `patches/meshcore/` — ordered patch queue applied to the submodule.
- `colorado/` — Colorado Mesh overlay files, fixtures, and notes.
- `scripts/` — wrapper scripts for patch and build workflows.
## Patch workflow
Initialize the submodule and apply patches:
```sh
git submodule update --init --recursive
bash scripts/apply-patches.sh
```
Develop firmware changes in `vendor/MeshCore`, commit them in that submodule worktree, then export the patch queue:
```sh
bash scripts/export-patches.sh origin/main
```
Representative companion build environments:
- `Heltec_v3_companion_radio_usb`
- `Heltec_v3_companion_radio_ble`
- `RAK_4631_companion_radio_usb`
- `RAK_4631_companion_radio_ble`

5
colorado/README.md Normal file
View File

@@ -0,0 +1,5 @@
# Colorado Mesh Firmware Bot Overlay
This directory holds Colorado Mesh source overlays, fixtures, and notes that are not part of the upstream MeshCore submodule.
Firmware changes that must live inside MeshCore for PlatformIO builds are developed in `vendor/MeshCore`, committed there temporarily, and exported into `patches/meshcore/` with `scripts/export-patches.sh`.

0
patches/meshcore/.gitkeep Normal file
View File

31
scripts/apply-patches.sh Executable file
View File

@@ -0,0 +1,31 @@
#!/usr/bin/env bash
set -euo pipefail
source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/meshcore-env.sh"
if [ ! -d "${MESHCORE_DIR}/.git" ] && [ ! -f "${MESHCORE_DIR}/.git" ]; then
echo "MeshCore submodule is missing at ${MESHCORE_DIR}" >&2
echo "Run: git submodule update --init --recursive" >&2
exit 1
fi
if [ -n "$(git -C "${MESHCORE_DIR}" status --porcelain)" ]; then
echo "MeshCore submodule has uncommitted or untracked changes; commit, export, or reset them before applying patches." >&2
exit 1
fi
shopt -s nullglob
patches=("${PATCH_DIR}"/*.patch)
shopt -u nullglob
if [ "${#patches[@]}" -eq 0 ]; then
echo "No MeshCore patches to apply. MeshCore commit: $(meshcore_commit)"
exit 0
fi
for patch in "${patches[@]}"; do
echo "Applying ${patch#${MESHCORE_FW_ROOT}/}"
git -C "${MESHCORE_DIR}" apply --index "$patch"
done
echo "Applied ${#patches[@]} MeshCore patch(es) to $(meshcore_commit)"

34
scripts/export-patches.sh Executable file
View File

@@ -0,0 +1,34 @@
#!/usr/bin/env bash
set -euo pipefail
source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/meshcore-env.sh"
base_ref="${1:-origin/main}"
if [ ! -d "${MESHCORE_DIR}/.git" ] && [ ! -f "${MESHCORE_DIR}/.git" ]; then
echo "MeshCore submodule is missing at ${MESHCORE_DIR}" >&2
echo "Run: git submodule update --init --recursive" >&2
exit 1
fi
if ! git -C "${MESHCORE_DIR}" rev-parse --verify "${base_ref}" >/dev/null 2>&1; then
echo "Base ref '${base_ref}' is not available in the MeshCore submodule." >&2
exit 1
fi
if [ -n "$(git -C "${MESHCORE_DIR}" status --porcelain)" ]; then
echo "MeshCore submodule has uncommitted or untracked changes; commit them in the submodule before exporting patches." >&2
exit 1
fi
commit_count="$(git -C "${MESHCORE_DIR}" rev-list --count "${base_ref}..HEAD")"
if [ "${commit_count}" -eq 0 ]; then
echo "No MeshCore commits to export after ${base_ref}."
exit 1
fi
mkdir -p "${PATCH_DIR}"
rm -f "${PATCH_DIR}"/*.patch
git -C "${MESHCORE_DIR}" format-patch --zero-commit --no-signature --output-directory "${PATCH_DIR}" "${base_ref}..HEAD" >/dev/null
echo "Exported ${commit_count} MeshCore patch(es) to ${PATCH_DIR#${MESHCORE_FW_ROOT}/}"

22
scripts/meshcore-env.sh Executable file
View File

@@ -0,0 +1,22 @@
#!/usr/bin/env bash
meshcore_fw_repo_root() {
local script_dir
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
(cd "${script_dir}/.." && pwd)
}
MESHCORE_FW_ROOT="${MESHCORE_FW_ROOT:-$(meshcore_fw_repo_root)}"
MESHCORE_DIR="${MESHCORE_DIR:-${MESHCORE_FW_ROOT}/vendor/MeshCore}"
PATCH_DIR="${PATCH_DIR:-${MESHCORE_FW_ROOT}/patches/meshcore}"
REPRESENTATIVE_ENVS=(
Heltec_v3_companion_radio_usb
Heltec_v3_companion_radio_ble
RAK_4631_companion_radio_usb
RAK_4631_companion_radio_ble
)
meshcore_commit() {
git -C "${MESHCORE_DIR}" rev-parse HEAD
}

1
vendor/MeshCore vendored Submodule

Submodule vendor/MeshCore added at 910b1bee5b