# 0.5.1 downlink investigation — 2026-10-04

Baseline: clean `main`, `7b5e6872fb2d807f0652148411b711c687a07789`. Initial `git status --short --branch` showed only `## main...origin/main`; `git log --oneline -20` was inspected; `git diff main...HEAD` was empty.

Branch: `codex/downlink-investigation-0.5.1`. Worktree: `/home/lighthouse/esp32-dev/.cache/downlink-investigation-0.5.1`. The original `/home/lighthouse/esp32-dev` remains on unchanged main. This matters: the user service and the installed Pi extension both refer to that original directory. Nothing was deployed. `device-gateway.service` remains active with PID `3848171`, started `2026-10-03 14:53:47 CST`; it was never restarted. Candidate firmware is in the worktree's `dist`, not production `dist`.

Implementation commit: `fef2777`. Transport tools/evidence: `862738b`. Physical checklist and Mac handoff: [physical-downlink-0.5.1-next-test.md](physical-downlink-0.5.1-next-test.md).

## Confirmed defects and the candidate

1. **CONFIRMED SOFTWARE BUG:** Pi `finalResponse(agent_end.messages)` was stored in the extension, sent on **`agent_settled`**, and stored as `PiSession.final_text`. `PiSession.public(include_history=True)` omitted it. `board_pi_snapshot_t`, the JSON decoder and `draw_pi()` had no final response field. With no `message_update`, DONE had no reply to draw. With tools, it could retain the last tool instead. Codex also retained a final response internally but omitted it from `AgentSession`.
2. **CONFIRMED SOFTWARE BUG:** Agent body text used M5GFX's default `Font0`, whose actual `font0_info` covers codepoints **0–255**. Chinese activity therefore rendered missing-glyph placeholders. The existing CN/TW fallback font was only selected for the voice confirmation modal. This explains Chinese invisibility, not disappearance of every English tool event or RUNNING timer.

**FIXED IN BRANCH:** add agent-neutral `final_text` to the device snapshot and common session model. The server redacts recognized credential forms before truncation, removes control characters, and sends at most **2048 UTF-8 bytes**, including a trailing ellipsis when shortened. It is a prefix of the authoritative final assistant response, not a copy of activity or session history. The full internal reply remains available to existing TTS handling. No credentials/configuration are added to the repository.

The firmware copies into a 2049-byte field without splitting a UTF-8 character. RUNNING uses the existing live text/tool view. DONE/FAILED with the explicit field uses the final response, even if late activity arrives. An explicit empty reply displays `No final response received.` beneath the terminal status. An absent field retains compatibility with older gateways. A new run clears the field; reconnect/target refresh returns it again. Final-only updates invalidate the display. Final display shows its first eight wrapped lines; live text retains the tail view. This is a bounded screen preview, not a new scrolling/history feature.

Agent body and Read Screen wrapping now use the existing simplified/traditional Chinese font, restoring the previous font afterward. Read Screen selects the same body/lines as the display. Emoji or arbitrary Unicode glyph coverage is not promised. New serial diagnostics log activity identity, acceptance, byte counts and page/modal, never text or audio.

## Field-level path

All Pi lifecycle/activity requests POST to `/integrations/pi/events`, authenticated with `x-device-lab-pi-token`. The SDK's actual installed `agent-session.js` and event types were checked against the handlers; no event-name assumption was carried over from earlier analysis.

| Meaning | Pi origin → extension payload | Registry storage | Device WebSocket payload | Firmware storage → UI |
|---|---|---|---|---|
| RUNNING | `agent_start` generates UUID `turn_id`; POST `event=agent_start`, `final_text=""`, `timestamp` | `state=RUNNING`, `turn_id`, `run_started_at`, zero elapsed, cleared final | `agent_state`: `state`, `elapsed_seconds`, metadata | `snapshot.state/elapsed_seconds` → `s_pi.state/elapsed_seconds/received_at_ms`; `draw_pi()` draws cyan RUNNING and locally advancing timer |
| Tool | `tool_execution_start` → `summarizeTool(toolName,args)` (Reading/Editing/Running/Searching); `tool_execution_end` → `Completed\n<tool>` or `Failed\n<tool>`; POST `event=activity`, `activity_type=tool`, `activity`, `tool_name` | `append_activity`: latest ≤480 UTF-8 bytes, type/time; history appends `\n[TOOL] ...\n`, tail ≤12 KiB | Immediate `agent_activity`: `activity_type`, `activity`, `updated_at`; subsequent `agent_state`: `latest_activity`, `recent_text`, `activity_type`, `activity_updated_at` | `board_agent_activity()` → `board_append_pi_activity()`, latest plus local tail, redraw pending; `draw_pi()` draws TOOL and last six lines of latest activity during RUNNING |
| Text | `message_update.assistantMessageEvent.type=text_delta`, `.delta` → `TextActivityBatcher`; POST `event=activity`, `activity_type=text`, `activity` | latest/type/time and concatenated bounded `recent_text` | Same immediate `agent_activity` shape; periodic state includes ≤4 KiB `recent_text` (generic Codex path ≤4000 bytes) | `board_agent_activity()` appends text; `board_set_pi_snapshot()` replaces tail from state; local tail ≤6 KiB, render source last 1400 bytes; last eight wrapped lines while RUNNING |
| Final assistant response | `agent_end.messages`: last assistant's text blocks, credential-redacted, ≤16000 codepoints; cached locally. **`agent_settled`** posts `event=agent_end`, `turn_id`, `final_text` once | Matching active turn stores `session.final_text` independently of activity; terminal deduplication | **Before: omitted. Candidate:** `PiSession.public()` → `agent_state.final_text`, ≤2048 bytes, explicit empty outside DONE/FAILED | `cloud_client.cpp` extracts `final_text` → `board_pi_snapshot_t.final_text` → `s_pi.has_final_text/final_text`; `agent_display_text()` chooses it for DONE/FAILED; `draw_pi()` and `read_screen_text()` share selection |
| DONE / FAILED | Cached `runFailed(messages)` detects assistant `stopReason=error/aborted`; settled POST sends `failed` | Transition only from RUNNING; `run_elapsed_seconds`; `terminal_notified`; one `pending_notification=done/failed` | `agent_state.state=DONE/FAILED`, elapsed, final, optional `notification`; periodic snapshots omit notification | State color green/red; final remains separate. `board_notify_pi()` handles vibration/tone independently of rendering |

Identity throughout: extension `ctx.sessionManager.getSessionId()` → event `session_id` → registry key → `target=agent:pi:<id>`. `name` and `cwd` come from the same extension context. `turn_id` validates/deduplicates terminal events in the registry; it is not a firmware field. `tool_name` is not rendered independently; the extension's summary is rendered. Failure state is not inferred from the final reply's wording.

Gateway route details: `pi_event()` applies the event; activity calls `push_selected_pi_activity()`, other events call `push_selected_pi_session()` → `agent_target_payload()` / `pi_target_payload()`. Only devices whose stored canonical target matches receive Pi events. Legacy `pi:<id>` selections normalize to `agent:pi:<id>`. `send_to_device()` serializes sends with a per-device lock, then awaits `websocket.send_json` with a 3-second timeout. `monitor_system()` also pushes agent snapshots every 2 seconds. Pi's adapter overlays live SDK catalog entries with registry state, with a registry fallback on catalog errors.

On device, the 16-KiB `WebSocketTextReceiver` reassembles complete text before JSON parsing. `agent_state` goes to `board_set_pi_snapshot()`; `agent_activity` requires matching selected agent/session before appending. Same-target snapshots preserve the page/modal. Real target changes select PI and cancel stale input/playback. Approval state may overlay `WAITING_APPROVAL` and its modal; it is separate from tool/text/final rendering.

Codex path: `turn/started` clears internal final; completed agent message items / deltas accumulate per turn; `turn/completed` prefers `phase=final_answer` and excludes commentary. Candidate `CodexAdapter.convert()` exposes bounded `final_text` via `AgentSession`; the common payload and firmware then use the same semantics. Existing terminal-to-speech handling is unchanged.

## Live process: what is and is not eliminated

| Hypothesis | Evidence and remaining boundary |
|---|---|
| A. No activity emitted | Registered extension handlers are exercised with real event shapes. Text timer, 120-codepoint threshold, tool start/end, session identity and settled final all pass. Thinking-only output is intentionally not activity. Production HTTP failures are still possible: optional POSTs have a 750-ms timeout, swallow errors and do not check HTTP status. |
| B. Gateway did not push | Authenticated in-process WebSocket tests and isolated TLS fake device receive RUNNING, tool, text and final in order. This proves these software paths with healthy connections, not last night's production delivery. |
| C. Wrong target/session | Canonical target selection and nonselected suppression pass. Actual firmware `board_agent_activity()` is compiled in a host harness: wrong agent/session is ignored; matching identity updates history and requests redraw. No original physical trace establishes the observed device's identity at the failure. |
| D. Redraw gating | Actual `pi_snapshot_changes_view()` passes final-only changes. Existing production redraw policy passes PI/overlay/non-PI checks; matching activity sets pending. `board_update()` coalesces visible updates at ≥250 ms and RUNNING ticks at 1 second. Thread scheduling, display transaction completion and overlay timing need hardware. |
| E. Not on PI page | Same-target updates deliberately do not force PI. Voice SENT feedback clears after about 1 second and selects PI. Approval/settings/voice/message/speech overlays hide Agent rendering. These conditions are real, but the original physical page/modal is unknown. |
| F. Font/rendering | Chinese Font0 defect reproduced using actual extracted `draw_pi()` and fixed. Linked CN/TW assets cover the checklist Chinese strings. English tools, actual brightness/layout and screen redraw remain physical checks. |

Batching: text flushes after **350 ms** or **120 codepoints**, before tool start, agent_end, settled and shutdown. Activity POSTs are fire-and-forget and may reach the server in a different order under delay; terminal no longer depends on their delivery. The extension reports terminal at settled, so retries/compaction/queued continuations can delay it after agent_end. New tests deliberately deliver late activity after DONE/FAILED and confirm the final response persists.

## Speech generation and transport

`SpeechManager.play()` cleans text, awaits the entire TTS PCM result, then sends start → chunks → end. Cloudflare Melo returns compressed audio as JSON/base64 or audio/mpeg; bounded input is converted by ffmpeg to **44100 Hz signed 16-bit little-endian mono**. The converter enforces input/output/time limits and cleans up cancelled processes. Unit tests now include a real MP3 encode/decode path, in addition to existing provider mocks and error/boundary checks.

| Item | Current behavior |
|---|---|
| PCM rate | 88,200 bytes/s = 705.6 kbit/s |
| Full chunk | 80 ms = 7056 raw bytes = 9408 base64 characters/bytes |
| JSON WebSocket text payload | 9502 bytes at sequence 0, 9503 at sequence 10, 9504 at sequence 100 (32-character speech ID, compact Starlette JSON) |
| WebSocket framing | TEXT, unmasked server frames; full chunk adds 4 framing bytes. TLS/TCP/IP overhead excluded from payload sizes. The test disables compression. |
| Pacing | Send one chunk, **await send completion**, then sleep `len(chunk)/88200`; last partial chunk sleeps its actual duration. `speech_end.chunks=ceil(pcm_bytes/7056)`. No catch-up loop. Approximately realtime, slightly slower including send/scheduler cost. |
| Server queue | Entire PCM retained in memory (bounded at 180 seconds). No per-chunk producer queue: one awaited chunk send per active device stream. Underlying TLS/kernel queued-frame maximum is not specified by this code. |
| ESP32 queue | 8 × 7056-byte frames in PSRAM, nominal 640 ms waiting PCM plus current consumer/DMA; physical occupancy unmeasured tonight. |
| Bandwidth/window comparison | Base64 alone 117,600 bytes/s; JSON about 118,800 bytes/s (~0.95 Mbit/s), plus framing/TLS/IP. Built TCP MSS=1440, receive window=65535 (~6.9 speech frames/~552 ms at nominal rate), mailbox=48. Nominal Wi-Fi support does not establish sustained goodput/RTT on this actual ESP32/hotspot/WAN path. |

The first draft smoke incorrectly required every **receive** interval ≥60 ms. The trace showed a delayed first chunk and two deliveries ~40 ms apart while later sends remained ~81 ms apart. The corrected test records the sender separately and validates receiver sequence/hash/queue rather than treating receive coalescing as a Gateway pacing defect. **No production pacing or networking patch was made.** A separate induced 200-ms send-stall test verifies no catch-up burst.

### Deterministic authenticated TLS result

The fake device continuously receives hello/state/activity/speech while the isolated gateway also pushes snapshots every 250 ms. A separate consumer simulates realtime drain with an eight-frame queue. Every PCM byte is checked by SHA-256; missing/duplicate sequence and wrong end counts are assertions. The recorder has negative tests for sequence loss/duplication. Queue measurements exclude the frame being consumed.

| Audio | Raw PCM bytes | Chunks | Max TEXT payload bytes | Start → end seconds | Effective PCM bytes/s | Effective text bytes/s | Queue peak |
|---|---:|---:|---:|---:|---:|---:|---:|
| 3 s | 264600 | 38 | 9503 | 3.047 | 86844 | 117063 | 1 |
| 10 s | 882000 | 125 | 9504 | 10.151 | 86889 | 117050 | 1 |
| 20 s | 1764000 | 250 | 9504 | 20.291 | 86937 | 117107 | 1 |

All cases: **0 missing, 0 duplicates, correct speech_end count, 0 server disconnects**. Send intervals 80.61–82.55 ms; receive intervals 39.71–113.50 ms. One expected client-initiated code-1000 close after all cases is recorded separately. This does not measure ESP32 receive timing or physical playback.

Full 697-message metadata trace, UTC/monotonic times, byte sizes, speech sequences, transfer statistics and disconnects: [downlink-0.5.1-transport.json](downlink-0.5.1-transport.json). No PCM/base64 or real credentials are stored.

Real provider smoke: Chinese 263482 PCM bytes / 2.987 s, English 237882 / 2.697 s, mixed 338548 / 3.838 s; synthesis plus conversion completed in 1.67–2.59 s. Real summary cases returned success/failed/partial without fallback. The script also ran STT on generated public phrases; the mixed transcription was imperfect (`pytest` became `Hattiest`). These are provider checks, not microphone/speaker checks. Inputs containing “37 tests passed” are explicit smoke fixtures, not claims about this investigation. Evidence: [downlink-0.5.1-provider.json](downlink-0.5.1-provider.json).

## Recent speech fixes, read against current code

| Commit | Classification | Evidence / current effect |
|---|---|---|
| `c284360` reassembly | **confirmed necessary** | Old code discarded any partial SDK text event, even if the full frame fit 16 KiB. Existing host regression reproduces a 9502-byte frame split 1294/7788/420 bytes, plus continuation/control/reset/error cases. Added to the standard test runner. This proves parser correctness, not physical stream health. |
| `be2ccec` 16-KiB TCP receive window / mailbox 14 | **plausible** | Capacity and link diagnostics are sensible; TCP streams do not require a window larger than a WebSocket message. No controlled physical before/after evidence proves these exact values necessary. Values were superseded by `7b5e687`; diagnostics remain. |
| `a51e13b` stream-relative idle timeout | **confirmed necessary** | Clear interleaving bug: old queue wait begins at t=0; new stream starts at t=2990 ms; old wait expires at t=3000 and could cancel a 10-ms-old stream. Current code checks start/last-accepted-PCM age under lock before expiring. The correctness race is distinct from whether 3 s is optimal on hardware. |
| `512516e` Wi-Fi PS none | **workaround** | Could reduce modem-sleep latency, but no reliable speaker proof. **It is no longer active:** `7b5e687` explicitly removes the `esp_wifi_set_ps(WIFI_PS_NONE)` call and log. Do not assume PS is disabled; read `ps=` on physical link logs. |
| `7b5e687` 65535 window / mailbox 48 | **possibly over-tuned** | Mailbox follows `ceil(window/MSS + 2)` at MSS 1440. More WAN data in flight may help, but exact need and memory tradeoff require physical evidence. Also removes the prior PS workaround. Left intact. |

No changes to `wifi_manager.cpp`, `sdkconfig.defaults`, `audio_backend.cpp`, TCP parameters, receive buffer capacity, timeout or I2S/DMA configuration were made tonight.

## Verification and limits

- Before patch: new authenticated final-response suite **5 failed / 1 passed**, missing `final_text` for Pi and Codex. Actual board render harness failed Chinese font and explicit-final assertions. After patch: final suite **6 passed**; actual render/Read Screen/identity/UTF-8/redraw harness passed.
- Complete `./scripts/test-server.sh`: **137 Python tests**, **18 Node tests** (Pi and Admin), six firmware host executables/harnesses, Gateway smoke and authenticated TLS voice smoke passed. One existing Starlette/AnyIO deprecation warning remains.
- Real provider speech smoke and deterministic TLS speech/downlink smoke passed. Delayed-send pacing and actual MP3 conversion tests passed.
- ESP-IDF 5.5.4 build and linked CN/TW glyph checks passed. Build is **SOFTWARE VERIFIED**, never physical proof. Candidate merged image: 2123456 bytes, SHA-256 `9826e18068006cb4b78df0026e60fbe1914e0a099a15215039ae32f4305f556d`. Version still `0.5.1`; distinguish the candidate by commit/artifact hash.
- `git diff --check` passed. Main and original production artifact were left unchanged. All tests used isolated runtime/credentials; real provider smoke only read existing provider credentials into memory.

Reproduce in the investigation worktree (socket-based tests require permission outside a socket-blocking sandbox):

```bash
./scripts/test-server.sh
server/.venv/bin/python scripts/smoke-downlink.py --output /tmp/downlink-transport.json
server/.venv/bin/python scripts/smoke-speech.py /tmp/downlink-provider.json
./scripts/build.sh m5stack-stopwatch
.cache/espressif/python_env/idf5.5_py3.11_env/bin/python firmware/tests/transcript_glyph_test.py firmware/build/device_lab.elf
git diff --check main...HEAD
```

## Final status taxonomy

| Area | Status | Boundary |
|---|---|---|
| Agent live activity | **CONFIRMED SOFTWARE BUG → FIXED IN BRANCH; SOFTWARE VERIFIED** | Chinese font defect fixed; emission/routing/identity/render selection tested. Original physical disappearance beyond Chinese glyphs is **UNKNOWN**; visibility/timer remain **REQUIRES PHYSICAL TEST**. |
| Agent final result | **CONFIRMED SOFTWARE BUG → FIXED IN BRANCH; SOFTWARE VERIFIED** | Explicit authoritative field and terminal rendering tested for Pi/Codex; actual screen remains **REQUIRES PHYSICAL TEST**. |
| TTS generation | **SOFTWARE VERIFIED** | Real public-text synthesis and compressed-audio conversion; no acoustic proof. |
| Gateway speech transport | **SOFTWARE VERIFIED** | Continuous local authenticated TLS receive, pacing/order/hash/count/queue; WAN-to-device behavior remains **REQUIRES PHYSICAL TEST**. |
| ESP32 speech receive | **REQUIRES PHYSICAL TEST** | Reassembler/protocol host tests pass, original receive failure cause **UNKNOWN**. |
| I2S playback | **REQUIRES PHYSICAL TEST** | No physical codec/DMA scheduling or underrun evidence tonight; cause **UNKNOWN**. |
| speaker audio | **REQUIRES PHYSICAL TEST** | No audible playback verification; cause **UNKNOWN**. |