# Network resilience 0.5.1

Branch: `codex/network-resilience-0.5.1`.
Base: `0f4f47ccbe79142a1b8cc324c62b856ef6433ff6`, fetched directly from
`lighthouse@claw.pangtiao.xyz:/home/lighthouse/esp32-dev` on 2026-10-04.
Isolated Mac checkout: `.cache/network-resilience-0.5.1`.

Before implementation, `git diff 6bffe57 0f4f47c -- firmware boards` was empty.
Gateway diff: `server/device_gateway/voice.py` added 17 content-free lifecycle
logging lines; `server/tests/test_voice_settings.py` added 27 diagnostic test
lines. No integration-source changes. Base commits are `40da493` and `0f4f47c`
above physically proven source `6bffe571b7c1b61cfdb168b0f7a03737bad394f1`.
The base worktree was clean; active Gateway PID was 419467, NRestarts=0, running
from `.cache/demo-text-voice-baseline-0.5.1`. Existing Speech branches and main
are not edited. Speech Output is FROZEN throughout this work.

## Configuration and recovery contract

The checked-in dependency lock and installed component header/source identify
`esp_websocket_client` 1.8.0 (ESP-IDF 5.5.4).
Source review: default Ping is 10s; default Pong deadline is 120s. The deadline
starts after the first unanswered Ping, and subsequent Pings do not reset it.
Configured Ping/Pong: 10s / 20s, with Pong-triggered disconnect enabled.
Silent half-open detection can therefore take roughly 30s plus scheduling or
blocking transport time; the preferred 3–5s recovery is after a detected
disconnect, not a guarantee for an undetected half-open link.

TCP keepalive is enabled at idle=30s, interval=10s, count=3. WebSocket Ping checks
the peer protocol endpoint; TCP keepalive checks the socket/path at the OS level.
Normal 10s Ping traffic generally prevents the longer TCP idle timer firing.
Application heartbeat remains 30s and is best effort.
Reconnect uses the component's fixed 1500ms delay, including clean server CLOSE.
Network operation timeout remains 10000ms; DNS/TLS/AP availability can dominate
wall-clock recovery. No reconnect loop or new retry protocol is introduced.

Each transport CONNECTED advances generation, sends a new authenticated hello,
and waits for hello_ack plus authoritative target/agent state before ONLINE.
Disconnect preserves Agent identity, selected target, final text, and last state.
Only the network footer becomes RECONNECTING. Queued controls and Approval
presentation/decisions are cleared; the Gateway must present current approval
again for fresh user confirmation. No history is replayed.

Voice Phase A cancels a broken turn and displays “Voice interrupted / Try again”.
Capture frames stay bound to their starting generation. No automatic Agent input
or old Voice request is replayed. Phase B PSRAM backup/replay is DEFERRED until
the basic physical network acceptance establishes whether it is needed.

## Validation

`GATEWAY_NODE_BIN=/home/lighthouse/.nvm/versions/node/v22.22.1/bin/node
./scripts/test-network-resilience.sh` passed: 87 Python tests, 18 Node tests,
five firmware host harnesses. The new harness compiles the actual production
WebSocket callbacks, JSON dispatcher, hello sender, and board disconnect/Voice
interruption functions against the installed ESP-IDF cJSON implementation.
Hardware/transport/display are substituted; physical rendering is not inferred.
New integration scenarios cover DONE, FAILED, and still-RUNNING while offline,
same selected session, final text/elapsed recovery, partial Voice cancellation
without STT or Agent input, and zero Agent input/close calls on disconnect.
Existing Approval reconnect regression also passes. No new protocol is needed.
ESP-IDF 5.5.4 build passed with installed esp_websocket_client 1.8.0.

The component emits some ERROR callbacks before assigning the abort error type;
DISCONNECTED/CLOSED error_handle and close code are logged too. A NONE in the
earlier ERROR callback is not relabeled by guesswork.
Only TCP_TRANSPORT initializes TLS/socket error fields in component 1.8.0.
Other event types log zero for those fields, avoiding uninitialized stack data.
This was discovered on the initial candidate connect and is regression-tested.

Physical environment remains phone hotspot → Cloudflare Tunnel → Lighthouse
Singapore. Speech/TTS/Read Screen/AUTO_REPLY/AUTO_SUMMARY/playback are excluded.

## Physical acceptance, 2026-10-04 (Asia/Shanghai)

**Network Resilience: PHYSICAL PASS for automatic recovery and Agent continuity.**
The preferred <=5s complete-state recovery was **not met** in these two trials;
do not present the reconnect-delay setting as an end-to-end latency guarantee.
No network provider, Wi-Fi power-save setting, Gateway deployment, or UI 0.6
merge was performed. Gateway stayed on baseline PID 419467, NRestarts=0.

Final firmware source: `c19afb3` (above implementation `c7e14d2`).
Merged artifact: 2,125,376 bytes, SHA-256
`4e93dcd7a2191fa732f7cf3c667554e516749dfe6a5b2dd624f04cb238fd2ea4`.
Application artifact: 2,059,840 bytes. Both uploads were verified by esptool.
The first candidate was replaced only to fix the uninitialized diagnostic fields.
Both flashing resets were intentional and excluded from unexpected reboot counts.

### Five-minute natural observation

Observation ran 22:01:41–22:06:42, 300.7s. Its first sample was during startup
authentication and is excluded from transient-outage counts. The first full
sync was 22:01:46; the initial transport→hello/target sync took 14.6s on this
network. These are startup observations, not recovery performance claims.

One natural WSS disconnect occurred; Wi-Fi did not disconnect. Device retained
the selected Pi session, DONE and the existing 50-byte final result.

| Disconnect 1 | Observed value |
|---|---|
| Device time | 22:06:00.141 |
| Reason | TCP transport / receive error; deeper cause unknown |
| esp_err / tls_err / sock_errno / close_code | 0 / 0 / 0 / 0 |
| RSSI at disconnect | -29 dBm |
| Gateway offline→online | 22:06:00.369→22:06:05.628, **5.259s** |
| Firmware disconnect→WSS CONNECTED | **4.938s** |
| Firmware disconnect→authoritative sync | **5.668s** |
| Generation / successful attempt | 1→2 / 1 |
| Snapshot restored | Same target, DONE, final_text (50 bytes) |
| Reboot / Agent cancellation | 0 / 0 |

### One controlled hotspot interruption + independent Agent task

One task was submitted at 22:07:41 to the already selected Pi session. It only
ran `sleep 20` and returned the specified test result. The user approved the
command on the device at about 22:08:32, then interrupted the hotspot once.
Device selection stayed `agent:pi:01a0fcd2-7b96-77fb-a57a-40bbfb9fe5da` throughout.

| Disconnect 2 | Observed value |
|---|---|
| Gateway offline→online | 22:08:35.286→22:09:18.573, **43.287s** including unavailable hotspot |
| First firmware disconnect | 22:08:42.079 |
| Reason | TCP after hotspot removal; Wi-Fi reason 200, then no-AP retries 201 |
| esp_err / tls_err / sock_errno / close_code | 0x801d / 76 / 0 / 0 |
| RSSI at disconnect | -127 sentinel (AP unavailable); recovered -40 dBm |
| Firmware disconnect→WSS / complete sync | **35.942s / 37.486s**, includes hotspot unavailable time |
| Wi-Fi IP restored→WSS / complete sync | **4.896s / 6.470s** |
| Generation / successful attempt | 2→3 / 21 (fixed-delay retries while hotspot unavailable) |

The cloud was RUNNING with device offline at 22:08:35 and reached DONE at
22:08:55 while the device remained offline. Its 24-byte final result matched
the expected answer. Reconnect delivered DONE, the same final result and
elapsed_seconds=73 (includes time waiting for human approval). Firmware applied
that snapshot at 22:09:19.564. The user confirmed automatic ONLINE / DONE and
the correct screen result, without a reboot or page switch. Pi event POSTs
continued returning 200; the observed window contained 47 successful event POSTs.

### Normal Voice and bounded scope

One recording after recovery: capture start=1, sent voice_start=1,
sent voice_end=1, transcript-ready=1, voice SENT=1, failed sends=0.
Pi then showed RUNNING→DONE and a 170-byte final result. The user confirmed
correct recognition, successful SEND, and Pi completion. **Normal Voice: PASS.**
This conclusion uses firmware lifecycle, Agent snapshots and human observations;
VOICE_RX/VOICE_STT module logs were not emitted in the captured Gateway journal.
No Voice content was printed in diagnostics or this report.

Voice Phase A software regression: PASS. A physical recording interruption was
not added; the single controlled interruption was used for Agent continuity.
Voice backup/replay: **DEFERRED**, no PSRAM utterance ownership added, no automatic
retry, no chunk resume. Existing live upload path remains the normal path.

Unexpected reboot count: **0** after the final intentional flashing reset.
Unexpected Agent cancellation count: **0** for the tested tasks.
Speech start count: **0**. Speech Output: **FROZEN**, mode OFF throughout.

Recommended next: **Wearable Shell 0.6a** is now eligible after the functional
physical pass. Keep a separate narrow recovery-latency follow-up for the observed
5.668s / 6.470s full-sync times; no further network probing or tuning was done in
this acceptance window.

Evidence (ignored local files) is in `.cache/physical-20261004/` in this worktree:
`serial.log`, `gateway-window.jsonl`, `idle-samples.jsonl`,
`agent-interruption-samples.jsonl`, `user-observations.json`, `result.json`,
and verified `flash-final.log`. Firmware artifacts are in `dist/m5stack-stopwatch/`.
Gateway runtime source is unchanged from `0f4f47c`; the Gateway diff only adds
the new regression test. Main, baseline and all Speech branches remain unchanged.