meshcore-sar_android/docs/voice-mode-technical.md

Voice Mode Technical Design

1. Overview

Voice mode uses a two-plane architecture:

• Control plane (text messages):
  • VE2: voice envelope announces voice availability in chat.
• Control plane (raw binary request):
  • Binary voice fetch request (same raw route as voice packets).
• Data plane (raw binary packets):
  • VoicePacket payload streamed via cmdSendRawData and received through pushRawData.

This design avoids broadcasting full voice payloads to channels/rooms. Chat carries only metadata; audio is fetched on demand when user presses play.

2. Key Modules

• lib/utils/voice_message_parser.dart
  • VoicePacket (legacy text + binary packet format)
  • VoiceEnvelope (VE2)
  • VoiceFetchRequest (binary)
• lib/screens/messages_tab.dart
  • Capture/encode voice, cache encoded packets, send envelope only
• lib/providers/voice_provider.dart
  • Reassembly/playback sessions
  • Outgoing session cache + deferred serving
• lib/providers/app_provider.dart
  • Incoming routing for VE2 and binary voice fetch requests
  • Handles raw packet ingestion
• lib/widgets/messages/voice_message_bubble.dart
  • Play behavior (immediate play if complete, otherwise fetch + auto-play)
• lib/providers/messages_provider.dart
  • Message-level voice detection (VE2 + legacy V:)
• lib/services/message_storage_service.dart
  • Persists isVoice and voiceId

3. Wire Formats

3.1 Voice Envelope (VE2)

Prefix: VE2: + colon-delimited compact payload (base36 numeric fields)

Fields:

• sid (string): base36 token for 32-bit session ID
• mode (base36): codec mode ID (VoicePacketMode.id)
• total (base36): packet count (1..255)
• durS (base36): estimated duration in seconds
• senderKey6 (string, 12 hex chars): sender public-key prefix (6 bytes)
• ts (base36): unix timestamp seconds

sid is base36 on wire and expands to 8-hex internally.

Compact format:

VE2:{sid}:{mode}:{total}:{durS}:{senderKey6}:{ts}

Example:

VE2:a:1:4:4:aabbccddeeff:s44we8

3.2 Voice Fetch Request (binary)

Binary payload format:

[magic=0x72][sid:4B][flags:1B][requesterKey6:6B][ts:4B][missingCount:1B][missingIndices...]

3.3 Raw Voice Packet (data plane)

Binary payload structure:

• Byte 0: magic 0x56 ('V')
• Bytes 1..4: session ID (4 bytes)
• Byte 5: mode ID
• Byte 6: packet index
• Byte 7: total packets
• Bytes 8..N: codec2 data

4. Outgoing Flow (Send)

1. Recorder captures PCM chunks.
2. Each chunk is codec2-encoded into VoicePacket objects.
3. Packets are cached in VoiceProvider outgoing cache (TTL 15 min).
4. Sender inserts local voice placeholder message (isVoice=true, voiceId=sessionId).
5. Sender sends one envelope (VE2) through normal message path:
   • channel/room: sendChannelMessage
   • direct: sendTextMessage
6. No raw audio packets are sent during initial send.

5. Incoming Routing

5.1 VE2 envelope received

AppProvider marks message as voice (isVoice, voiceId) and adds it to chat.

5.2 Binary voice fetch request received

AppProvider treats it as control-plane only:

• request is not added to chat
• validates requester prefix match against sender metadata
• resolves requester contact via key prefix
• calls voiceProvider.serveSessionTo(...)

5.3 Raw packet received (pushRawData)

AppProvider.onRawDataReceived parses VoicePacket binary and appends to session in VoiceProvider.

6. Play / Fetch Behavior

In VoiceMessageBubble:

• If session already complete: play immediately.
• If incomplete/missing:
  1. Resolve sender contact (message sender prefix or VE2.senderKey6 fallback)
  2. Send direct binary fetch request
  3. Show requesting state in UI
  4. Auto-play when session becomes complete

If sender cannot be resolved or request cannot be sent, bubble remains and shows: "Voice unavailable right now".

7. Outgoing Cache Details

VoiceProvider outgoing cache:

• key: sessionId
• value: encoded packet list + cached timestamp
• TTL: 15 minutes (_outgoingSessionTtl)
• eviction: lazy (on cache access/add/serve)

Serving prerequisites:

• session exists in cache
• sendRawPacketCallback configured
• requester has direct path (outPathLen >= 0)

8. Persistence

MessageStorageService now stores and restores:

• isVoice
• voiceId

This ensures envelope messages remain voice bubbles across app restart.

9. Validation and Safety

Parser validation enforces:

• strict hex lengths for IDs and key prefixes
• valid mode range
• valid packet counts and duration bounds
• compact base36 numeric fields in envelope/request
• Binary request flags specify all or missing indices.
• Request payload includes requesterKey6 to resolve return route.

10. Transmit Time Estimate (UI)

Voice bubbles and Message Technical Details show an estimated transmit time (~... tx).

The estimate is airtime-based (LoRa packet model), not file-duration-only:

• Source inputs:
  • packetCount and durationMs from VE2 envelope, or
  • numeric envelope values decoded from base36
  • actual received VoicePacket.codec2Data.length bytes when local session packets exist
  • pathLen from message metadata
  • current radio params from deviceInfo: radioBw, radioSf, radioCr
• Per-packet payload model:
  • meshHeader(2) + pathLen + voiceHeader(8) + codec2Bytes
• LoRa airtime:
  • standard symbol-time formula (preamble + payload symbols)
• Mesh pacing/hops:
  • multiplied by (1 + airtimeBudgetFactor) where default factor is 1.0
  • multiplied by hop count (pathLen + 1)
• Total estimate:
  • sum over all packets

BW handling:

• If radioBw is index 0..9, app maps it to Hz (7.8k .. 500k)
• If radioBw > 1000, it is treated as Hz directly

Fallback defaults are used when radio params are unavailable: SF10, BW250kHz, CR5.

11. Operational Constraints

• No firmware changes required.
• On-demand fetch works only if sender app is online and has cached session.
• Raw return path needs a currently valid direct route to requester.
• Voice capture is available on iOS and Android (Platform.isIOS || Platform.isAndroid).

11.1 Raw Binary Routing Semantics

• Voice payload packets use companion command CMD_SEND_RAW_DATA (25 / 0x19).
• Companion push back to the app is PUSH_CODE_RAW_DATA (0x84).
• Over-the-air packet type for this flow is PAYLOAD_TYPE_RAW_CUSTOM (0x0F).
• This flow is direct-route only, not flood/broadcast:
  • it is sent to one destination path;
  • only nodes on that path relay it;
  • it is not received by everyone in the mesh.

12. Backward Compatibility

• Legacy V: text packet parsing is still supported.
• Message voice detection accepts VE2 and legacy V: formats.

13. High-Level Sequence

sequenceDiagram
  participant A as Sender App
  participant M as Mesh Chat
  participant B as Receiver App

  A->>A: Record + encode voice packets
  A->>A: Cache session packets (TTL 15m)
  A->>M: Send VE2 envelope
  M->>B: Deliver VE2
  B->>B: Render voice bubble (metadata only)
  B->>A: Send binary fetch request on Play
  A->>B: Stream raw VoicePacket packets
  B->>B: Reassemble session
  B->>B: Auto-play when complete