Add resend mechanism for media

This commit is contained in:
Janez T
2026-03-01 19:29:09 +01:00
parent a0c4cdeede
commit 790c205541
3 changed files with 144 additions and 62 deletions

View File

@@ -5,6 +5,8 @@ Flutter app for Search and Rescue operations over a [MeshCore](https://github.co
## Features ## Features
- **Messaging** — send/receive messages to contacts and channels - **Messaging** — send/receive messages to contacts and channels
- **Voice messages** — push-to-talk voice over the mesh using [Codec2](https://github.com/drowe67/codec2) ultra-low-bitrate speech compression (7003200 bps); fragments are streamed on demand via direct BLE links without requiring firmware changes
- **Image messages** — send photos over the mesh as AVIF-compressed fragments; images are fetched on demand when the recipient taps the placeholder; configurable resolution (64 / 128 / 256 px), compression level, and colour/grayscale mode
- **Contacts** — track team members, repeaters, and rooms with live telemetry - **Contacts** — track team members, repeaters, and rooms with live telemetry
- **SAR Markers** — drop emoji-coded location pins (person found, fire, staging area) via chat messages - **SAR Markers** — drop emoji-coded location pins (person found, fire, staging area) via chat messages
- **Map** — view team positions and markers on OpenStreetMap / topo / satellite; tiles cached for offline use - **Map** — view team positions and markers on OpenStreetMap / topo / satellite; tiles cached for offline use
@@ -70,10 +72,12 @@ cd ios && pod deintegrate && pod install && cd ..
| Platform | Permissions | | Platform | Permissions |
|----------|-------------| |----------|-------------|
| iOS | Bluetooth, Location (when in use) | | iOS | Bluetooth, Location (when in use), Microphone (voice), Photo Library / Camera (images) |
| Android | `BLUETOOTH_SCAN`, `BLUETOOTH_CONNECT`, `ACCESS_FINE_LOCATION`, `INTERNET` | | Android | `BLUETOOTH_SCAN`, `BLUETOOTH_CONNECT`, `ACCESS_FINE_LOCATION`, `INTERNET`, `RECORD_AUDIO`, `READ_MEDIA_IMAGES` |
## SAR Marker format ## Wire protocols
### SAR Marker
``` ```
S:<emoji>:<lat>,<lon> S:<emoji>:<lat>,<lon>
@@ -81,6 +85,40 @@ S:<emoji>:<lat>,<lon>
Examples: `S:🧑:46.0569,14.5058` · `S:🔥:46.057,14.506` · `S:🏕️:46.0571,14.506` Examples: `S:🧑:46.0569,14.5058` · `S:🔥:46.057,14.506` · `S:🏕️:46.0571,14.506`
### Voice messages (on-demand, direct contacts only)
Control plane (text):
```
VE1:{sid}:{mode}:{total}:{durationMs}:{senderKey6}:{ts}:{ver} ← envelope
VR1:{sid}:a:{requesterKey6}:{ts}:{ver} ← fetch request
```
Data plane (binary via `cmdSendRawData` / `pushRawData`, ≤160 bytes/packet):
```
[0x56 'V'][sessionId:4B][mode:1B][idx:1B][total:1B][Codec2 frame...]
```
Codec2 modes: 700C / 1200 / 1300 / 1400 / 1600 / 2400 / 3200 bps.
### Image messages (on-demand, direct contacts only)
Control plane (text):
```
IE1:{sid}:{fmt}:{total}:{w}:{h}:{bytes}:{senderKey6}:{ts}:{ver} ← envelope
IR1:{sid}:a:{requesterKey6}:{ts}:{ver} ← fetch request
```
Data plane (binary via `cmdSendRawData` / `pushRawData`, ≤160 bytes/packet):
```
[0x49 'I'][sessionId:4B][fmt:1B][idx:1B][total:1B][AVIF fragment...]
```
Images are compressed to AVIF (configurable up to 256×256, grayscale by default). At 152 bytes of payload per fragment a typical image takes 720 fragments — comparable to a short voice clip.
## Architecture ## Architecture
``` ```

View File

@@ -22,15 +22,20 @@ pixels are fetched on demand when the user taps the image bubble.
- `fragmentImage()` — split compressed bytes into packets - `fragmentImage()` — split compressed bytes into packets
- `reassembleImage()` — join received fragments into bytes - `reassembleImage()` — join received fragments into bytes
- `lib/screens/messages_tab.dart` - `lib/screens/messages_tab.dart`
- Pick image, compress to 128×128 grayscale AVIF, cache, send envelope - Pick image from gallery or camera, compress with user settings, cache, send envelope
- `lib/providers/image_provider.dart` - `lib/providers/image_provider.dart`
- Reassembly sessions, outgoing cache, deferred serving - Reassembly sessions, outgoing cache, deferred serving
- Outgoing sessions also registered as complete incoming sessions for immediate local display
- `lib/providers/app_provider.dart` - `lib/providers/app_provider.dart`
- Incoming routing for `IE1`, `IR1`, binary `0x49` packets - Incoming routing for `IE1`, `IR1`, binary `0x49` packets
- `lib/widgets/messages/image_message_bubble.dart` - `lib/widgets/messages/image_message_bubble.dart`
- Placeholder with tap-to-load; progress ring during fetch; full image view - Square cover thumbnail (up to 256 px); tap-to-load for received images;
progress ring during fetch; full-screen `InteractiveViewer` on tap
- `lib/services/image_codec_service.dart` - `lib/services/image_codec_service.dart`
- `ImageCodecService.compress()``dart:ui` resize + grayscale + AVIF encode - `ImageCodecService.compress()`aspect-ratio-preserving resize + optional
grayscale + AVIF encode via `flutter_avif`
- `lib/services/image_preferences.dart`
- `ImagePreferences` — persists max size, compression level, grayscale toggle
## 3. Wire Formats ## 3. Wire Formats
@@ -40,17 +45,17 @@ Prefix: `IE1:` + colon-delimited payload
Fields: Fields:
| Field | Type | Description | | Field | Type | Description |
|-------------|--------|------------------------------------------| |--------------|--------|------------------------------------------------|
| `sid` | string | 8 hex chars (4 bytes), session ID | | `sid` | string | 8 hex chars (4 bytes), session ID |
| `fmt` | int | `ImageFormat.id` (0 = AVIF, 1 = JPEG) | | `fmt` | int | `ImageFormat.id` (0 = AVIF, 1 = JPEG) |
| `total` | int | Fragment count (1..255) | | `total` | int | Fragment count (1..255) |
| `w` | int | Image width (pixels) | | `w` | int | Actual image width after compression (pixels) |
| `h` | int | Image height (pixels) | | `h` | int | Actual image height after compression (pixels) |
| `bytes` | int | Total compressed size in bytes | | `bytes` | int | Total compressed size in bytes |
| `senderKey6`| string | 12 hex chars (6 bytes sender prefix) | | `senderKey6` | string | 12 hex chars (6 bytes sender prefix) |
| `ts` | int | Unix timestamp (seconds) | | `ts` | int | Unix timestamp (seconds) |
| `ver` | int | Protocol version (currently `1`) | | `ver` | int | Protocol version (currently `1`) |
Compact format: Compact format:
@@ -58,12 +63,15 @@ Compact format:
IE1:{sid}:{fmt}:{total}:{w}:{h}:{bytes}:{senderKey6}:{ts}:{ver} IE1:{sid}:{fmt}:{total}:{w}:{h}:{bytes}:{senderKey6}:{ts}:{ver}
``` ```
Example: Example (256×171 landscape image, 14 fragments):
```text ```text
IE1:deadbeef:0:7:128:128:1050:aabbccddeeff:1700000000:1 IE1:deadbeef:0:14:256:171:2100:aabbccddeeff:1700000000:1
``` ```
Note: `w` and `h` reflect the actual post-compression dimensions, which preserve
the source aspect ratio (contain within the configured max size).
### 3.2 Image Fetch Request (`IR1`) ### 3.2 Image Fetch Request (`IR1`)
Same structure as `VR1`: Same structure as `VR1`:
@@ -72,11 +80,11 @@ Same structure as `VR1`:
IR1:{sid}:{want}:{requesterKey6}:{ts}:{ver} IR1:{sid}:{want}:{requesterKey6}:{ts}:{ver}
``` ```
| Field | Value | | Field | Value |
|----------------|--------| |------------------|--------------------------|
| `want` | `a` (= "all fragments") | | `want` | `a` (= "all fragments") |
| `requesterKey6`| 12 hex chars | | `requesterKey6` | 12 hex chars |
| `ver` | `1` | | `ver` | `1` |
### 3.3 Raw Image Packet (data plane) ### 3.3 Raw Image Packet (data plane)
@@ -93,49 +101,72 @@ Header is 8 bytes — identical layout to `VoicePacket`.
## 4. Compression Pipeline ## 4. Compression Pipeline
1. Source image (any format) is decoded via `dart:ui.instantiateImageCodec` `ImageCodecService.compress(rawBytes, {maxDimension, compression, grayscale})`:
with `targetWidth: 128, targetHeight: 128`.
2. RGBA pixels exported via `image.toByteData(format: rawRgba)`.
3. Converted to grayscale (luminance `0.299R + 0.587G + 0.114B`) in-place.
4. Encoded to AVIF via `flutter_avif.encodeAvif()` with:
- `quality: 60` (CQ scale — lower = better quality)
- `speed: 8` (fast encode)
5. Fragmented at 152 bytes per packet.
### Expected sizes (128×128 grayscale AVIF) 1. **Probe** — decode source image at original resolution to read `srcW × srcH`.
2. **Contain** — compute `dstW × dstH` that fits within `maxDimension × maxDimension`
while preserving aspect ratio. Images smaller than the cap are not upscaled.
3. **Decode**`dart:ui.instantiateImageCodec` with `targetWidth: dstW, targetHeight: dstH`.
4. **RGBA export**`image.toByteData(format: rawRgba)`.
5. **Grayscale** *(optional, default on)* — luminance `0.299R + 0.587G + 0.114B`
applied in-place; if disabled, colour pixels are passed through unchanged.
6. **PNG re-encode**`ui.ImageDescriptor.raw``toByteData(format: png)`.
Required because `encodeAvif()` takes an encoded image, not raw RGBA.
7. **AVIF encode**`flutter_avif.encodeAvif(pngBytes, maxQuantizer, minQuantizer, speed: 8)`.
`compression` (1090) maps to libavif CQ scale:
`maxQ = round(compression/100 × 63)`, `minQ = round(maxQ × 0.65)`.
8. **Fragment**`fragmentImage()` at 152 bytes/packet (≤255 packets).
| Quality | Approx size | Fragments | Returns `({Uint8List bytes, int width, int height})` with the actual compressed dimensions.
|---------|-------------|-----------|
| 40 | 400800 B | 36 |
| 60 | 6001400 B | 410 |
| 80 | 10002500 B | 717 |
Quality 60 targets 49 fragments — comparable to a 10-second voice session. ### User-configurable settings (`ImagePreferences`)
| Setting | Key | Default | Range / Options |
|--------------|--------------------|---------|-----------------|
| Max size | `image_max_size` | 256 | 64 / 128 / 256 |
| Compression | `image_quality` | 90 | 1090 |
| Grayscale | `image_grayscale` | true | true / false |
### Expected sizes (grayscale AVIF, compression 90)
| Max size | Typical size | Fragments |
|----------|--------------|-----------|
| 64×64 | 100400 B | 13 |
| 128×128 | 300900 B | 26 |
| 256×256 | 7003000 B | 520 |
Non-square images produce fewer fragments than the square equivalent because
only the shorter axis is padded — no cropping occurs.
## 5. Outgoing Flow (Send) ## 5. Outgoing Flow (Send)
1. User picks image from gallery or camera (`image_picker`). 1. User picks image from gallery or camera (`image_picker`).
2. `ImageCodecService.compress()` produces small grayscale AVIF bytes. 2. `ImagePreferences` is read: max size, compression, grayscale.
3. `fragmentImage()` splits bytes into `ImagePacket` list (≤255 fragments). 3. `ImageCodecService.compress()` returns `(bytes, width, height)`.
4. Fragments cached in `ImageProvider` (TTL 15 min). 4. `fragmentImage()` splits bytes into `ImagePacket` list (≤255 fragments).
5. `ImageEnvelope` is sent via normal message path: 5. `ImageProvider.cacheOutgoingSession()` stores fragments in both the outgoing
cache (for serving to peers) and the incoming session map (so the local bubble
renders the image immediately without a fetch round-trip).
6. `ImageEnvelope` built with actual `width`/`height` from step 3.
7. Envelope sent via normal message path:
- Channel: `sendChannelMessage` - Channel: `sendChannelMessage`
- Direct: `sendTextMessage` - Direct: `sendTextMessage`
6. Local placeholder message added with `IE1:` text and `deliveryStatus.sending`. 8. Local placeholder message added (`IE1:` text, `deliveryStatus.sending`).
## 6. Incoming Flow (Receive) ## 6. Incoming Flow (Receive)
### 6.1 `IE1` envelope received ### 6.1 `IE1` envelope received
`AppProvider` calls `imageProvider.registerEnvelope()` and adds the message to chat. The bubble shows a grey placeholder with a download icon. `AppProvider` calls `imageProvider.registerEnvelope()` and adds the message to
chat. The bubble shows a grey square placeholder with a download icon.
### 6.2 `IR1` request received ### 6.2 `IR1` request received
`AppProvider` treats it as control-plane only (not added to chat): `AppProvider` treats it as control-plane only (not added to chat):
- Validates requester key prefix matches sender metadata. - Validates requester key prefix.
- Resolves requester contact. - Resolves requester contact.
- Calls `imageProvider.serveSessionTo()` which streams all fragments. - Calls `imageProvider.serveSessionTo()` which streams all cached fragments.
### 6.3 Raw packet received (`pushRawData`, magic `0x49`) ### 6.3 Raw packet received (`pushRawData`, magic `0x49`)
@@ -152,14 +183,27 @@ automatically rebuilds with the full image.
- TTL: 15 minutes - TTL: 15 minutes
- eviction: lazy on access - eviction: lazy on access
When `cacheOutgoingSession()` is called it also writes all fragments into
`_sessions[sessionId]`, so the sender sees the image immediately in the bubble
(no tap-to-load required for own messages).
## 8. Display ## 8. Display
`ImageMessageBubble`: `ImageMessageBubble` (max width 256 px):
- **Complete session**: 128×128 `AvifImage.memory()` widget; tap → full-screen `InteractiveViewer`. - **Complete session**: `AspectRatio(1.0)``AvifImage.memory(fit: cover)`
- **Incomplete/missing**: grey placeholder with download icon; tap → sends IR1. square thumbnail; tap → full-screen `InteractiveViewer` with fade transition.
- **Loading**: circular progress with `received/total` count. - **Incomplete/missing**: grey square placeholder with download icon;
- **Error**: "Image unavailable right now" text. tap → sends `IR1` fetch request.
- **Loading**: circular progress indicator showing `received/total` count.
- **Error**: broken-image icon.
Status line below thumbnail:
- Sender: `🖼️ {w}×{h} AVIF · {n} seg`
- Receiver (complete): `🖼️ {w}×{h} AVIF`
- Receiver (pending): `🖼️ Tap to load · {w}×{h}`
- Loading: `📥 Loading… {received}/{total}`
## 9. Persistence ## 9. Persistence
@@ -168,15 +212,14 @@ automatically rebuilds with the full image.
- Incoming: fragment list serialized as base64 binary packets. - Incoming: fragment list serialized as base64 binary packets.
- Outgoing: fragment list + envelope text + `cachedAt` timestamp. - Outgoing: fragment list + envelope text + `cachedAt` timestamp.
- Expired outgoing sessions (> 15 min) are not restored. - Expired outgoing sessions (> 15 min) are not restored on startup.
## 10. Operational Constraints ## 10. Operational Constraints
- No firmware changes required (reuses `cmdSendRawData` / `pushRawData`). - No firmware changes required (reuses `cmdSendRawData` / `pushRawData`).
- On-demand fetch works only if sender app is online and has cached session. - On-demand fetch works only if sender app is online and has cached session.
- Raw return path requires a valid direct route to requester. - Raw return path requires a valid direct route to requester.
- Image compression uses `flutter_avif` — the `encodeAvif()` top-level - Available on iOS and Android (`image_picker` + `flutter_avif`).
function must be available (verify against installed package version).
## 11. High-Level Sequence ## 11. High-Level Sequence
@@ -186,14 +229,15 @@ sequenceDiagram
participant M as Mesh Chat participant M as Mesh Chat
participant B as Receiver App participant B as Receiver App
A->>A: Pick + compress image (128×128 grayscale AVIF) A->>A: Pick image (gallery or camera)
A->>A: Compress: contain resize → grayscale → PNG → AVIF
A->>A: Fragment into ≤152B packets A->>A: Fragment into ≤152B packets
A->>A: Cache fragments (TTL 15m) A->>A: Cache outgoing + populate local session (immediate display)
A->>M: Send IE1 envelope A->>M: Send IE1 envelope (actual w×h, fragment count)
M->>B: Deliver IE1 M->>B: Deliver IE1
B->>B: Render placeholder bubble B->>B: Render grey placeholder bubble
B->>A: Send IR1 request on Tap B->>A: Tap → send IR1 fetch request
A->>B: Stream binary ImagePackets A->>B: Stream binary ImagePackets
B->>B: Reassemble fragments B->>B: Reassemble fragments
B->>B: Display AVIF image B->>B: Display AVIF image (cover thumbnail)
``` ```

View File

@@ -180,7 +180,7 @@ Parser validation enforces:
- No firmware changes required. - No firmware changes required.
- On-demand fetch works only if sender app is online and has cached session. - On-demand fetch works only if sender app is online and has cached session.
- Raw return path needs a currently valid direct route to requester. - Raw return path needs a currently valid direct route to requester.
- Voice capture in UI is currently iOS-only (`MessagesTab._voiceSupported`). - Voice capture is available on iOS and Android (`Platform.isIOS || Platform.isAndroid`).
## 11. Backward Compatibility ## 11. Backward Compatibility