mirror of
https://github.com/dz0ny/meshcore-sar.git
synced 2026-08-11 16:30:28 +00:00
Add resend mechanism for media
This commit is contained in:
44
README.md
44
README.md
@@ -5,6 +5,8 @@ Flutter app for Search and Rescue operations over a [MeshCore](https://github.co
|
|||||||
## Features
|
## Features
|
||||||
|
|
||||||
- **Messaging** — send/receive messages to contacts and channels
|
- **Messaging** — send/receive messages to contacts and channels
|
||||||
|
- **Voice messages** — push-to-talk voice over the mesh using [Codec2](https://github.com/drowe67/codec2) ultra-low-bitrate speech compression (700–3200 bps); fragments are streamed on demand via direct BLE links without requiring firmware changes
|
||||||
|
- **Image messages** — send photos over the mesh as AVIF-compressed fragments; images are fetched on demand when the recipient taps the placeholder; configurable resolution (64 / 128 / 256 px), compression level, and colour/grayscale mode
|
||||||
- **Contacts** — track team members, repeaters, and rooms with live telemetry
|
- **Contacts** — track team members, repeaters, and rooms with live telemetry
|
||||||
- **SAR Markers** — drop emoji-coded location pins (person found, fire, staging area) via chat messages
|
- **SAR Markers** — drop emoji-coded location pins (person found, fire, staging area) via chat messages
|
||||||
- **Map** — view team positions and markers on OpenStreetMap / topo / satellite; tiles cached for offline use
|
- **Map** — view team positions and markers on OpenStreetMap / topo / satellite; tiles cached for offline use
|
||||||
@@ -70,10 +72,12 @@ cd ios && pod deintegrate && pod install && cd ..
|
|||||||
|
|
||||||
| Platform | Permissions |
|
| Platform | Permissions |
|
||||||
|----------|-------------|
|
|----------|-------------|
|
||||||
| iOS | Bluetooth, Location (when in use) |
|
| iOS | Bluetooth, Location (when in use), Microphone (voice), Photo Library / Camera (images) |
|
||||||
| Android | `BLUETOOTH_SCAN`, `BLUETOOTH_CONNECT`, `ACCESS_FINE_LOCATION`, `INTERNET` |
|
| Android | `BLUETOOTH_SCAN`, `BLUETOOTH_CONNECT`, `ACCESS_FINE_LOCATION`, `INTERNET`, `RECORD_AUDIO`, `READ_MEDIA_IMAGES` |
|
||||||
|
|
||||||
## SAR Marker format
|
## Wire protocols
|
||||||
|
|
||||||
|
### SAR Marker
|
||||||
|
|
||||||
```
|
```
|
||||||
S:<emoji>:<lat>,<lon>
|
S:<emoji>:<lat>,<lon>
|
||||||
@@ -81,6 +85,40 @@ S:<emoji>:<lat>,<lon>
|
|||||||
|
|
||||||
Examples: `S:🧑:46.0569,14.5058` · `S:🔥:46.057,14.506` · `S:🏕️:46.0571,14.506`
|
Examples: `S:🧑:46.0569,14.5058` · `S:🔥:46.057,14.506` · `S:🏕️:46.0571,14.506`
|
||||||
|
|
||||||
|
### Voice messages (on-demand, direct contacts only)
|
||||||
|
|
||||||
|
Control plane (text):
|
||||||
|
|
||||||
|
```
|
||||||
|
VE1:{sid}:{mode}:{total}:{durationMs}:{senderKey6}:{ts}:{ver} ← envelope
|
||||||
|
VR1:{sid}:a:{requesterKey6}:{ts}:{ver} ← fetch request
|
||||||
|
```
|
||||||
|
|
||||||
|
Data plane (binary via `cmdSendRawData` / `pushRawData`, ≤160 bytes/packet):
|
||||||
|
|
||||||
|
```
|
||||||
|
[0x56 'V'][sessionId:4B][mode:1B][idx:1B][total:1B][Codec2 frame...]
|
||||||
|
```
|
||||||
|
|
||||||
|
Codec2 modes: 700C / 1200 / 1300 / 1400 / 1600 / 2400 / 3200 bps.
|
||||||
|
|
||||||
|
### Image messages (on-demand, direct contacts only)
|
||||||
|
|
||||||
|
Control plane (text):
|
||||||
|
|
||||||
|
```
|
||||||
|
IE1:{sid}:{fmt}:{total}:{w}:{h}:{bytes}:{senderKey6}:{ts}:{ver} ← envelope
|
||||||
|
IR1:{sid}:a:{requesterKey6}:{ts}:{ver} ← fetch request
|
||||||
|
```
|
||||||
|
|
||||||
|
Data plane (binary via `cmdSendRawData` / `pushRawData`, ≤160 bytes/packet):
|
||||||
|
|
||||||
|
```
|
||||||
|
[0x49 'I'][sessionId:4B][fmt:1B][idx:1B][total:1B][AVIF fragment...]
|
||||||
|
```
|
||||||
|
|
||||||
|
Images are compressed to AVIF (configurable up to 256×256, grayscale by default). At 152 bytes of payload per fragment a typical image takes 7–20 fragments — comparable to a short voice clip.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -22,15 +22,20 @@ pixels are fetched on demand when the user taps the image bubble.
|
|||||||
- `fragmentImage()` — split compressed bytes into packets
|
- `fragmentImage()` — split compressed bytes into packets
|
||||||
- `reassembleImage()` — join received fragments into bytes
|
- `reassembleImage()` — join received fragments into bytes
|
||||||
- `lib/screens/messages_tab.dart`
|
- `lib/screens/messages_tab.dart`
|
||||||
- Pick image, compress to 128×128 grayscale AVIF, cache, send envelope
|
- Pick image from gallery or camera, compress with user settings, cache, send envelope
|
||||||
- `lib/providers/image_provider.dart`
|
- `lib/providers/image_provider.dart`
|
||||||
- Reassembly sessions, outgoing cache, deferred serving
|
- Reassembly sessions, outgoing cache, deferred serving
|
||||||
|
- Outgoing sessions also registered as complete incoming sessions for immediate local display
|
||||||
- `lib/providers/app_provider.dart`
|
- `lib/providers/app_provider.dart`
|
||||||
- Incoming routing for `IE1`, `IR1`, binary `0x49` packets
|
- Incoming routing for `IE1`, `IR1`, binary `0x49` packets
|
||||||
- `lib/widgets/messages/image_message_bubble.dart`
|
- `lib/widgets/messages/image_message_bubble.dart`
|
||||||
- Placeholder with tap-to-load; progress ring during fetch; full image view
|
- Square cover thumbnail (up to 256 px); tap-to-load for received images;
|
||||||
|
progress ring during fetch; full-screen `InteractiveViewer` on tap
|
||||||
- `lib/services/image_codec_service.dart`
|
- `lib/services/image_codec_service.dart`
|
||||||
- `ImageCodecService.compress()` — `dart:ui` resize + grayscale + AVIF encode
|
- `ImageCodecService.compress()` — aspect-ratio-preserving resize + optional
|
||||||
|
grayscale + AVIF encode via `flutter_avif`
|
||||||
|
- `lib/services/image_preferences.dart`
|
||||||
|
- `ImagePreferences` — persists max size, compression level, grayscale toggle
|
||||||
|
|
||||||
## 3. Wire Formats
|
## 3. Wire Formats
|
||||||
|
|
||||||
@@ -40,17 +45,17 @@ Prefix: `IE1:` + colon-delimited payload
|
|||||||
|
|
||||||
Fields:
|
Fields:
|
||||||
|
|
||||||
| Field | Type | Description |
|
| Field | Type | Description |
|
||||||
|-------------|--------|------------------------------------------|
|
|--------------|--------|------------------------------------------------|
|
||||||
| `sid` | string | 8 hex chars (4 bytes), session ID |
|
| `sid` | string | 8 hex chars (4 bytes), session ID |
|
||||||
| `fmt` | int | `ImageFormat.id` (0 = AVIF, 1 = JPEG) |
|
| `fmt` | int | `ImageFormat.id` (0 = AVIF, 1 = JPEG) |
|
||||||
| `total` | int | Fragment count (1..255) |
|
| `total` | int | Fragment count (1..255) |
|
||||||
| `w` | int | Image width (pixels) |
|
| `w` | int | Actual image width after compression (pixels) |
|
||||||
| `h` | int | Image height (pixels) |
|
| `h` | int | Actual image height after compression (pixels) |
|
||||||
| `bytes` | int | Total compressed size in bytes |
|
| `bytes` | int | Total compressed size in bytes |
|
||||||
| `senderKey6`| string | 12 hex chars (6 bytes sender prefix) |
|
| `senderKey6` | string | 12 hex chars (6 bytes sender prefix) |
|
||||||
| `ts` | int | Unix timestamp (seconds) |
|
| `ts` | int | Unix timestamp (seconds) |
|
||||||
| `ver` | int | Protocol version (currently `1`) |
|
| `ver` | int | Protocol version (currently `1`) |
|
||||||
|
|
||||||
Compact format:
|
Compact format:
|
||||||
|
|
||||||
@@ -58,12 +63,15 @@ Compact format:
|
|||||||
IE1:{sid}:{fmt}:{total}:{w}:{h}:{bytes}:{senderKey6}:{ts}:{ver}
|
IE1:{sid}:{fmt}:{total}:{w}:{h}:{bytes}:{senderKey6}:{ts}:{ver}
|
||||||
```
|
```
|
||||||
|
|
||||||
Example:
|
Example (256×171 landscape image, 14 fragments):
|
||||||
|
|
||||||
```text
|
```text
|
||||||
IE1:deadbeef:0:7:128:128:1050:aabbccddeeff:1700000000:1
|
IE1:deadbeef:0:14:256:171:2100:aabbccddeeff:1700000000:1
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Note: `w` and `h` reflect the actual post-compression dimensions, which preserve
|
||||||
|
the source aspect ratio (contain within the configured max size).
|
||||||
|
|
||||||
### 3.2 Image Fetch Request (`IR1`)
|
### 3.2 Image Fetch Request (`IR1`)
|
||||||
|
|
||||||
Same structure as `VR1`:
|
Same structure as `VR1`:
|
||||||
@@ -72,11 +80,11 @@ Same structure as `VR1`:
|
|||||||
IR1:{sid}:{want}:{requesterKey6}:{ts}:{ver}
|
IR1:{sid}:{want}:{requesterKey6}:{ts}:{ver}
|
||||||
```
|
```
|
||||||
|
|
||||||
| Field | Value |
|
| Field | Value |
|
||||||
|----------------|--------|
|
|------------------|--------------------------|
|
||||||
| `want` | `a` (= "all fragments") |
|
| `want` | `a` (= "all fragments") |
|
||||||
| `requesterKey6`| 12 hex chars |
|
| `requesterKey6` | 12 hex chars |
|
||||||
| `ver` | `1` |
|
| `ver` | `1` |
|
||||||
|
|
||||||
### 3.3 Raw Image Packet (data plane)
|
### 3.3 Raw Image Packet (data plane)
|
||||||
|
|
||||||
@@ -93,49 +101,72 @@ Header is 8 bytes — identical layout to `VoicePacket`.
|
|||||||
|
|
||||||
## 4. Compression Pipeline
|
## 4. Compression Pipeline
|
||||||
|
|
||||||
1. Source image (any format) is decoded via `dart:ui.instantiateImageCodec`
|
`ImageCodecService.compress(rawBytes, {maxDimension, compression, grayscale})`:
|
||||||
with `targetWidth: 128, targetHeight: 128`.
|
|
||||||
2. RGBA pixels exported via `image.toByteData(format: rawRgba)`.
|
|
||||||
3. Converted to grayscale (luminance `0.299R + 0.587G + 0.114B`) in-place.
|
|
||||||
4. Encoded to AVIF via `flutter_avif.encodeAvif()` with:
|
|
||||||
- `quality: 60` (CQ scale — lower = better quality)
|
|
||||||
- `speed: 8` (fast encode)
|
|
||||||
5. Fragmented at 152 bytes per packet.
|
|
||||||
|
|
||||||
### Expected sizes (128×128 grayscale AVIF)
|
1. **Probe** — decode source image at original resolution to read `srcW × srcH`.
|
||||||
|
2. **Contain** — compute `dstW × dstH` that fits within `maxDimension × maxDimension`
|
||||||
|
while preserving aspect ratio. Images smaller than the cap are not upscaled.
|
||||||
|
3. **Decode** — `dart:ui.instantiateImageCodec` with `targetWidth: dstW, targetHeight: dstH`.
|
||||||
|
4. **RGBA export** — `image.toByteData(format: rawRgba)`.
|
||||||
|
5. **Grayscale** *(optional, default on)* — luminance `0.299R + 0.587G + 0.114B`
|
||||||
|
applied in-place; if disabled, colour pixels are passed through unchanged.
|
||||||
|
6. **PNG re-encode** — `ui.ImageDescriptor.raw` → `toByteData(format: png)`.
|
||||||
|
Required because `encodeAvif()` takes an encoded image, not raw RGBA.
|
||||||
|
7. **AVIF encode** — `flutter_avif.encodeAvif(pngBytes, maxQuantizer, minQuantizer, speed: 8)`.
|
||||||
|
`compression` (10–90) maps to libavif CQ scale:
|
||||||
|
`maxQ = round(compression/100 × 63)`, `minQ = round(maxQ × 0.65)`.
|
||||||
|
8. **Fragment** — `fragmentImage()` at 152 bytes/packet (≤255 packets).
|
||||||
|
|
||||||
| Quality | Approx size | Fragments |
|
Returns `({Uint8List bytes, int width, int height})` with the actual compressed dimensions.
|
||||||
|---------|-------------|-----------|
|
|
||||||
| 40 | 400–800 B | 3–6 |
|
|
||||||
| 60 | 600–1400 B | 4–10 |
|
|
||||||
| 80 | 1000–2500 B | 7–17 |
|
|
||||||
|
|
||||||
Quality 60 targets 4–9 fragments — comparable to a 10-second voice session.
|
### User-configurable settings (`ImagePreferences`)
|
||||||
|
|
||||||
|
| Setting | Key | Default | Range / Options |
|
||||||
|
|--------------|--------------------|---------|-----------------|
|
||||||
|
| Max size | `image_max_size` | 256 | 64 / 128 / 256 |
|
||||||
|
| Compression | `image_quality` | 90 | 10–90 |
|
||||||
|
| Grayscale | `image_grayscale` | true | true / false |
|
||||||
|
|
||||||
|
### Expected sizes (grayscale AVIF, compression 90)
|
||||||
|
|
||||||
|
| Max size | Typical size | Fragments |
|
||||||
|
|----------|--------------|-----------|
|
||||||
|
| 64×64 | 100–400 B | 1–3 |
|
||||||
|
| 128×128 | 300–900 B | 2–6 |
|
||||||
|
| 256×256 | 700–3000 B | 5–20 |
|
||||||
|
|
||||||
|
Non-square images produce fewer fragments than the square equivalent because
|
||||||
|
only the shorter axis is padded — no cropping occurs.
|
||||||
|
|
||||||
## 5. Outgoing Flow (Send)
|
## 5. Outgoing Flow (Send)
|
||||||
|
|
||||||
1. User picks image from gallery or camera (`image_picker`).
|
1. User picks image from gallery or camera (`image_picker`).
|
||||||
2. `ImageCodecService.compress()` produces small grayscale AVIF bytes.
|
2. `ImagePreferences` is read: max size, compression, grayscale.
|
||||||
3. `fragmentImage()` splits bytes into `ImagePacket` list (≤255 fragments).
|
3. `ImageCodecService.compress()` returns `(bytes, width, height)`.
|
||||||
4. Fragments cached in `ImageProvider` (TTL 15 min).
|
4. `fragmentImage()` splits bytes into `ImagePacket` list (≤255 fragments).
|
||||||
5. `ImageEnvelope` is sent via normal message path:
|
5. `ImageProvider.cacheOutgoingSession()` stores fragments in both the outgoing
|
||||||
|
cache (for serving to peers) and the incoming session map (so the local bubble
|
||||||
|
renders the image immediately without a fetch round-trip).
|
||||||
|
6. `ImageEnvelope` built with actual `width`/`height` from step 3.
|
||||||
|
7. Envelope sent via normal message path:
|
||||||
- Channel: `sendChannelMessage`
|
- Channel: `sendChannelMessage`
|
||||||
- Direct: `sendTextMessage`
|
- Direct: `sendTextMessage`
|
||||||
6. Local placeholder message added with `IE1:` text and `deliveryStatus.sending`.
|
8. Local placeholder message added (`IE1:` text, `deliveryStatus.sending`).
|
||||||
|
|
||||||
## 6. Incoming Flow (Receive)
|
## 6. Incoming Flow (Receive)
|
||||||
|
|
||||||
### 6.1 `IE1` envelope received
|
### 6.1 `IE1` envelope received
|
||||||
|
|
||||||
`AppProvider` calls `imageProvider.registerEnvelope()` and adds the message to chat. The bubble shows a grey placeholder with a download icon.
|
`AppProvider` calls `imageProvider.registerEnvelope()` and adds the message to
|
||||||
|
chat. The bubble shows a grey square placeholder with a download icon.
|
||||||
|
|
||||||
### 6.2 `IR1` request received
|
### 6.2 `IR1` request received
|
||||||
|
|
||||||
`AppProvider` treats it as control-plane only (not added to chat):
|
`AppProvider` treats it as control-plane only (not added to chat):
|
||||||
|
|
||||||
- Validates requester key prefix matches sender metadata.
|
- Validates requester key prefix.
|
||||||
- Resolves requester contact.
|
- Resolves requester contact.
|
||||||
- Calls `imageProvider.serveSessionTo()` which streams all fragments.
|
- Calls `imageProvider.serveSessionTo()` which streams all cached fragments.
|
||||||
|
|
||||||
### 6.3 Raw packet received (`pushRawData`, magic `0x49`)
|
### 6.3 Raw packet received (`pushRawData`, magic `0x49`)
|
||||||
|
|
||||||
@@ -152,14 +183,27 @@ automatically rebuilds with the full image.
|
|||||||
- TTL: 15 minutes
|
- TTL: 15 minutes
|
||||||
- eviction: lazy on access
|
- eviction: lazy on access
|
||||||
|
|
||||||
|
When `cacheOutgoingSession()` is called it also writes all fragments into
|
||||||
|
`_sessions[sessionId]`, so the sender sees the image immediately in the bubble
|
||||||
|
(no tap-to-load required for own messages).
|
||||||
|
|
||||||
## 8. Display
|
## 8. Display
|
||||||
|
|
||||||
`ImageMessageBubble`:
|
`ImageMessageBubble` (max width 256 px):
|
||||||
|
|
||||||
- **Complete session**: 128×128 `AvifImage.memory()` widget; tap → full-screen `InteractiveViewer`.
|
- **Complete session**: `AspectRatio(1.0)` → `AvifImage.memory(fit: cover)`
|
||||||
- **Incomplete/missing**: grey placeholder with download icon; tap → sends IR1.
|
square thumbnail; tap → full-screen `InteractiveViewer` with fade transition.
|
||||||
- **Loading**: circular progress with `received/total` count.
|
- **Incomplete/missing**: grey square placeholder with download icon;
|
||||||
- **Error**: "Image unavailable right now" text.
|
tap → sends `IR1` fetch request.
|
||||||
|
- **Loading**: circular progress indicator showing `received/total` count.
|
||||||
|
- **Error**: broken-image icon.
|
||||||
|
|
||||||
|
Status line below thumbnail:
|
||||||
|
|
||||||
|
- Sender: `🖼️ {w}×{h} AVIF · {n} seg`
|
||||||
|
- Receiver (complete): `🖼️ {w}×{h} AVIF`
|
||||||
|
- Receiver (pending): `🖼️ Tap to load · {w}×{h}`
|
||||||
|
- Loading: `📥 Loading… {received}/{total}`
|
||||||
|
|
||||||
## 9. Persistence
|
## 9. Persistence
|
||||||
|
|
||||||
@@ -168,15 +212,14 @@ automatically rebuilds with the full image.
|
|||||||
|
|
||||||
- Incoming: fragment list serialized as base64 binary packets.
|
- Incoming: fragment list serialized as base64 binary packets.
|
||||||
- Outgoing: fragment list + envelope text + `cachedAt` timestamp.
|
- Outgoing: fragment list + envelope text + `cachedAt` timestamp.
|
||||||
- Expired outgoing sessions (> 15 min) are not restored.
|
- Expired outgoing sessions (> 15 min) are not restored on startup.
|
||||||
|
|
||||||
## 10. Operational Constraints
|
## 10. Operational Constraints
|
||||||
|
|
||||||
- No firmware changes required (reuses `cmdSendRawData` / `pushRawData`).
|
- No firmware changes required (reuses `cmdSendRawData` / `pushRawData`).
|
||||||
- On-demand fetch works only if sender app is online and has cached session.
|
- On-demand fetch works only if sender app is online and has cached session.
|
||||||
- Raw return path requires a valid direct route to requester.
|
- Raw return path requires a valid direct route to requester.
|
||||||
- Image compression uses `flutter_avif` — the `encodeAvif()` top-level
|
- Available on iOS and Android (`image_picker` + `flutter_avif`).
|
||||||
function must be available (verify against installed package version).
|
|
||||||
|
|
||||||
## 11. High-Level Sequence
|
## 11. High-Level Sequence
|
||||||
|
|
||||||
@@ -186,14 +229,15 @@ sequenceDiagram
|
|||||||
participant M as Mesh Chat
|
participant M as Mesh Chat
|
||||||
participant B as Receiver App
|
participant B as Receiver App
|
||||||
|
|
||||||
A->>A: Pick + compress image (128×128 grayscale AVIF)
|
A->>A: Pick image (gallery or camera)
|
||||||
|
A->>A: Compress: contain resize → grayscale → PNG → AVIF
|
||||||
A->>A: Fragment into ≤152B packets
|
A->>A: Fragment into ≤152B packets
|
||||||
A->>A: Cache fragments (TTL 15m)
|
A->>A: Cache outgoing + populate local session (immediate display)
|
||||||
A->>M: Send IE1 envelope
|
A->>M: Send IE1 envelope (actual w×h, fragment count)
|
||||||
M->>B: Deliver IE1
|
M->>B: Deliver IE1
|
||||||
B->>B: Render placeholder bubble
|
B->>B: Render grey placeholder bubble
|
||||||
B->>A: Send IR1 request on Tap
|
B->>A: Tap → send IR1 fetch request
|
||||||
A->>B: Stream binary ImagePackets
|
A->>B: Stream binary ImagePackets
|
||||||
B->>B: Reassemble fragments
|
B->>B: Reassemble fragments
|
||||||
B->>B: Display AVIF image
|
B->>B: Display AVIF image (cover thumbnail)
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -180,7 +180,7 @@ Parser validation enforces:
|
|||||||
- No firmware changes required.
|
- No firmware changes required.
|
||||||
- On-demand fetch works only if sender app is online and has cached session.
|
- On-demand fetch works only if sender app is online and has cached session.
|
||||||
- Raw return path needs a currently valid direct route to requester.
|
- Raw return path needs a currently valid direct route to requester.
|
||||||
- Voice capture in UI is currently iOS-only (`MessagesTab._voiceSupported`).
|
- Voice capture is available on iOS and Android (`Platform.isIOS || Platform.isAndroid`).
|
||||||
|
|
||||||
## 11. Backward Compatibility
|
## 11. Backward Compatibility
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user