# Device Lab protocol 0.6.0 设备通过 `wss://gw.pangtiao.men/device/ws` 连接。连接后的第一帧必须是： ```json { "type": "hello", "device_id": "esp32s3-28848543a200", "device_token": "", "board": "m5stack-stopwatch", "firmware": "0.6.0", "capabilities": ["agent_target", "approval", "voice_input", "settings", "speech-settings"] } ``` Since 0.6.3 the negotiated navigator capabilities also include `history`; a hello that omits it simply gets no History UI and no History frames. Gateway 校验 token 和已绑定的 device id，失败时以 WebSocket policy error 关闭。成功后依次发送 `hello_ack`、最新 `server_snapshot` 和当前 `target_state`（server）或 `agent_state`（Pi/Codex）。 ## Gateway → device `server_snapshot` 每 2 秒主动推送。Gateway 只保留最新一帧以及计算网速所需的上一帧。 ```json { "type": "server_snapshot", "hostname": "lighthouse", "cpu_percent": 23.4, "load_1m": 0.42, "load_5m": 0.31, "memory_used": 4294967296, "memory_total": 8589934592, "memory_percent": 52.5, "swap_percent": 7.1, "disk_used": 987654321, "disk_total": 1234567890, "disk_percent": 80.0, "network_rx_bytes": 123456789, "network_tx_bytes": 98765432, "network_rx_bps": 1834212, "network_tx_bps": 328192, "uptime": 123456, "timestamp": "2026-10-02T12:00:00+00:00", "host_state": "Active", "codex_quota_valid": true, "codex_remaining_percent": 72, "codex_resets_at": 1759678800, "codex_window_minutes": 300 } ``` `host_state` 是 Gateway 从同一帧派生的 `Offline` / `Idle` / `Active` / `Busy` 分类，Watch 的 `Host` 行直接使用它。阈值：CPU ≥ 70% 或 MEM ≥ 90% 为 Busy，CPU ≥ 20% 为 Active，其余为 Idle；Gateway 与设备都带 5% hysteresis， 避免阈值附近每 2 秒抖动。 `codex_*` 是 Gateway 缓存的 Codex 账户 rate-limit 视图。它不新增帧、 不新增请求、设备也无权请求刷新：只是同一帧多带几个小字段。Gateway 在 Codex initialize 成功后调用一次原生的 `account/rateLimits/read`，并在收到 `account/rateLimits/updated`（sparse）后异步重读一次完整快照；同一时间最 多一个 refresh。设备只收到剩余百分比、重置时间和窗口时长，绝不包含 accountId、plan 或 credits。 Server target： ```json {"type":"target_state","target":"server"} ``` Agent target（Pi/Codex）： ```json { "type": "agent_state", "agent": "pi", "target": "agent:pi:", "model": "provider/model-id", "effort": null, "session_id": "", "name": "esp32-dev", "cwd": "/home/lighthouse/esp32-dev", "state": "RUNNING", "elapsed_seconds": 151, "waiting_reason": null, "activity_type": "text", "latest_activity": "The target selection request...", "recent_text": "...recent UTF-8-safe output tail...", "activity_updated_at": "2026-10-02T12:00:00+00:00" } ``` Gateway 为每个 Pi session 保留最多 12 KiB 的内存 recent history；Admin/API 默认只返回 480 B preview，设备取得 target 时返回最近 4 KiB。历史不落盘。仅选择了对应 session 的设备会收到该 session 的增量： ```json { "type": "agent_activity", "agent": "pi", "target": "agent:pi:", "session_id": "", "activity_type": "tool", "activity": "Running\npytest -q", "updated_at": "2026-10-02T12:00:01+00:00" } ``` 真实 `RUNNING -> DONE/FAILED` 时，Gateway 只在首次 terminal transition 的 `agent_state` 中附带 `"notification":"done"` 或 `"failed"`。重复的 `agent_end/agent_settled` 不再附带通知。开发自测通过受 Cloudflare Access 管理边界保护的 `POST /devices/{device_id}/test-notification` 发送： ```json {"kind":"done"} ``` 设备 WSS 对应收到 `{"type":"notification_test","kind":"done"}`；`failed` 同理。 `display_message` 保留第一阶段格式。 ## Device → Gateway 设备继续发送 `heartbeat` 和实体按键的 `button_event`。Button A/B 的本地页面切换不依赖 Gateway。 ## Pi Extension → Gateway Extension 向 loopback `POST /integrations/pi/events` 发送 `pi_session_state`，并用独立的 `X-Device-Lab-Pi-Token` header 认证。lifecycle 事件为 `session_start`、`session_info_changed`、`agent_start`、`agent_end`、`session_shutdown`。实时内容使用 `event: "activity"`，并带 `activity_type: "text" | "tool"`、`activity` 及可选 `tool_name`。Pi 端使用正式 `message_update/text_delta` 和 `tool_execution_start/end`，文本按 350 ms 或 120 字符批量发送，工具参数只保留脱敏后的短摘要；审批决定使用下述独立的 Approval control 协议与本机 Unix socket。 ## Approval control (0.4.0) After authenticated `hello`, Gateway presents the oldest still-pending approval, regardless of the selected target. Approval does not change the stored target. ```json {"type":"approval_request","approval_id":"opaque-unique-id","agent":"codex","session_id":"thread-id","project_id":"project-id","kind":"command","title":"Run command?","detail":"touch /tmp/proof","choices":["approve","reject"],"created_at":"2026-10-03T01:00:00Z","expires_at":null,"state":"PENDING","expires_in":null} ``` Device click/release response: ```json {"type":"approval_response","approval_id":"opaque-unique-id","decision":"approve"} ``` B uses `reject`. This is a distinct event; `button_event` never approves anything. Gateway requires PENDING, valid decision, and the exact ID currently presented to this authenticated WebSocket connection. Invalid/stale/duplicate responses are ignored and warned. Admin and device race through the same atomic resolve method. All devices that were shown a completed request receive: ```json {"type":"approval_cancelled","approval_id":"opaque-unique-id","state":"APPROVED"} ``` Terminal `state` is APPROVED / REJECTED / CANCELLED. EXPIRED is accepted only for compatibility; no default path produces it. NO DECISION = KEEP WAITING. The same terminal envelope removes a presentation on every device; APPROVED/REJECTED show one second of feedback. EXPIRED/CANCELLED dismiss immediately. Then Gateway sends the next FIFO pending item. Device holds only a tiny transient queue, never durable history. Disconnect clears the device overlay and outbound decision; reconnect only replays requests still PENDING in Gateway. Codex disconnect cancels requests from the old App Server connection and never replays their responses on the new connection. The unified AgentSession state includes WAITING_APPROVAL and waiting_reason. This is a derived overlay: removing it reveals the most recent real lifecycle state. ## Voice (0.6.0) All messages use the same authenticated `/device/ws`; no upload endpoint. ```json {"type":"voice_start","voice_id":"random-id","sample_rate":44100,"bits":16,"channels":1,"max_recording_ms":20000} {"type":"voice_chunk","voice_id":"random-id","sequence":0,"audio":""} {"type":"voice_status","voice_id":"random-id"} {"type":"voice_end","voice_id":"random-id","recorded_duration_ms":4000,"total_bytes":176400} {"type":"voice_cancel","voice_id":"random-id"} {"type":"voice_decision","voice_id":"random-id","decision":"send"} ``` `voice_decision` also accepts `cancel` and `retry`. Gateway pins the selected agent/session and device before capture. Recording is decoupled from upload: the device records the full clip to PSRAM first, then uploads it in acknowledged chunks, so a weak link stalls the upload without ever truncating or losing the recording. * `voice_chunk` is acknowledged with `voice_ack` (`voice_id`, `next_sequence`, `state`, plus a legacy `sequence` alias) where `next_sequence` is the next sequence the Gateway expects. The device only sends the next chunk after that acknowledgement, so no audio is lost silently. * Replaying an already accepted chunk is idempotent: the Gateway re-sends the current `voice_ack` and does not append the same audio twice. * A chunk whose sequence is ahead of the expected one is refused outright; the recording is cancelled rather than splicing a hole into the audio. * After a transport drop the Gateway holds the unfinished recording and reports `voice_state: SUSPENDED`. On reconnect the device queries `voice_status` with the same `voice_id`, receives the authoritative `voice_ack`, and resumes from that sequence. Any other stale message from the replacement connection is ignored; only an explicit `voice_status` resumes. * A device whose bounded upload window elapsed while the recording is still retained asks `voice_decision: retry`; the Gateway re-arms the upload window and answers with the breakpoint. The device never restarts the recording and never reports success for a partial upload. * Recording length (`recorded_duration_ms`, capped at 20 s) is validated separately from the upload window, so a slow resume is not mistaken for a long recording. If the device never returns, the upload window expires and the Gateway releases the recording without transcribing it. * The 20-second server capture timer never starts STT by itself: it only suspends an unfinished upload. Only an explicit `voice_end` starts STT. When supplied, `total_bytes` must be an integer equal to the received PCM length; mismatch cancels the request. Older clients may omit this field. * After integrity validation and transition to `TRANSCRIBING`, Gateway sends `{"type":"voice_end_ack","voice_id":"random-id","state":"TRANSCRIBING"}`. Firmware must use this explicit terminal upload acknowledgement, not the last chunk ACK or a successful local END write, to release retained upload audio. If the END or its acknowledgement is lost, query `voice_status` on reconnect and retry END when still recording. Status in `TRANSCRIBING` or `AWAITING_CONFIRMATION` sends the same `voice_end_ack` shape with current state. Duplicate END re-sends this acknowledgement without restarting STT; when the transcript is ready, both status and duplicate END re-deliver it. Neither acknowledgement nor transcript delivery submits input: explicit SEND is required. Target/context changes, approval and cancellation invalidate the request; invalidated requests cannot acknowledge END or re-deliver a transcript. * Replacement sockets preserve the request. Old-socket cleanup cannot suspend a request already adopted by the new socket. During STT, results are retained across disconnect and re-delivered by `voice_status`, without re-running STT or automatically submitting input to the Agent. IDs cannot be replayed to bind a new recording on a new connection. Gateway replies with `voice_state` (RECORDING / SUSPENDED / TRANSCRIBING / SENT / CANCELLED), `voice_ack`, `voice_transcript` (AWAITING_CONFIRMATION, agent, session_id, name, transcript), or `voice_error` (FAILED and a safe message). The device never sends transcript text back: SEND refers only to the already-bound voice ID. A device that does not declare the resume capability simply never sends `voice_status` and behaves as before. Limits and failure semantics are in [voice and settings](voice-input-design.md). ## Session settings (0.4.0) ```json {"type":"settings_get","request_id":"random-id"} {"type":"settings_apply","request_id":"random-id","target":"agent:codex:thread-id","settings":{"model":"model-from-catalog","effort":"effort-from-metadata"}} {"type":"settings_cancel","request_id":"random-id"} ``` Gateway replies `settings_state` / `settings_updated` with the request ID, pinned target, agent, session_id, name, model, effort, available_models, available_efforts. Each model carries id/name/default_effort/available_efforts and optional is_default. Quick catalog is deduplicated and capped at six. `settings_error` retains the request ID and safe error message; Gateway values remain authoritative. Target switch, approval and disconnect invalidate the presentation. A setting cannot apply to a different target, and duplicate apply is ignored. Admin uses `GET/POST /api/agents/{agent}/sessions/{id}/settings`; POST accepts only model/effort. No approval timeout settings are exposed. Busy sessions return 409. Legacy stored `pi:` targets are interpreted as `agent:pi:`. Firmware 0.4.0 also understands legacy `target_state pi:...` and `pi_activity`. Gateway emits only the unified Agent messages; upgrade older firmware before device testing. ## Speech playback (0.5.0) A hello without `speech_playback` remains valid, including Firmware 0.4.0. Gateway never sends `speech_*` playback to such devices. Speech Mode defaults OFF; modes are stored per device + target outside the repo. Explicit screen reading/test works in OFF. All following device requests use the existing authenticated WSS connection. Since 0.6.3 the preference split is explicit. A device may advertise `speech-settings` without `speech_playback`: it receives `speech_settings` and can save a mode with `speech_settings_set`, but never receives `speech_start`/ `speech_chunk`/`speech_end` and cannot enable playback by saving a preference. Firmware 0.6.3 advertises `speech-settings` and not `speech_playback`; Speech playback, Read Screen, AUTO_REPLY and AUTO_SUMMARY audio all remain FROZEN. Firmware 0.8.0 restores playback and also advertises `speech-buffer-v1`. Gateway → Device: ```json {"type":"speech_settings","target":"agent:pi:s","mode":"OFF","request_id":null} {"type":"speech_start","speech_id":"unique-id","target":"agent:pi:s","sample_rate":44100,"bits":16,"channels":1,"source":"reply","text":"Bounded preview.","pcm_bytes":7056,"pcm_sha256":"<64 hex characters>"} {"type":"speech_chunk","speech_id":"unique-id","sequence":0,"audio":""} {"type":"speech_end","speech_id":"unique-id","chunks":1} {"type":"speech_cancel","speech_id":"unique-id"} {"type":"speech_error","message":"Speech unavailable"} ``` `source` is screen/reply/summary. PCM is 44100 Hz, signed 16-bit little endian mono. Each chunk holds at most 7056 bytes (80 ms / 9408 base64 characters); sequences start at zero. Frames stay below 10 KiB, within the 16 KiB device receive buffer. Gateway sends at audio rate to legacy devices. With `speech-buffer-v1`, it sends ahead at most 64 frames beyond the acknowledged codec progress. The 0.8.0 device queues 128 frames in PSRAM, prebuffers 13 frames (1.04 seconds), and buffers complete utterances of at most 50 frames (4 seconds) before playback. Overflow or sequence error aborts. `speech_end.chunks` must match the received count and does not stop queued playback early. Stale IDs are ignored. Cancel clears queued PCM. Device → Gateway: ```json {"type":"speech_settings_get","request_id":"read-mode"} {"type":"speech_settings_set","request_id":"set-mode","target":"agent:pi:s","mode":"MANUAL"} {"type":"speech_request","request_id":"read-screen","target":"agent:pi:s","source":"screen","text":"Codex。任务完成。Tests passed."} {"type":"speech_stop","speech_id":"unique-id"} {"type":"speech_progress","speech_id":"unique-id","played_chunks":8} ``` Setting/request identities are bounded, single-use per connection. `target` must match the current target for mutations/screen reading, protecting against races. Screen text is at most 1200 UTF-8 bytes. A speech_stop with no ID also cancels pending synthesis (used when local recording/Settings/Approval preempts before speech_start). A stop carrying an old ID cannot cancel a replacement stream. Approval and voice_start also cancel server-side work independently. Disconnect invalidates pending work. Progress is monotonic, bound to the current speech ID, and cannot exceed the sent chunk count. It is acknowledged every eight codec frames. Receive stalls expire after three seconds when prebuffering or out of queued audio; a missing credit expires the Gateway sender after five seconds. Byte count and SHA-256 are checked before accepting the end frame; codec completion logs the written count/hash. Quiet Cloudflare PCM is boosted by at most 4x toward RMS 2500, with peak ceiling 26000; loud PCM and silence remain unchanged. Admin: `POST /api/devices/{id}/speech` accepts target/mode; `POST /api/devices/{id}/speech-test` accepts only target and speaks a fixed sentence. Existing Cloudflare Access protection applies. Model/effort settings are unchanged. See [Speech design](spoken-agent-design.md) for UI, provider and final-reply semantics. ## 0.6.0 additive UTC field Authenticated `hello_ack` may include integer `server_time_ms` (UTC Unix milliseconds) and `timezone_offset_minutes` (-720..840). Old clients ignore these optional fields; CONTROL_VERSION remains 0.5.0. New clients accept finite integer timestamps in 2024..2100, record the receive-time monotonic base, write UTC to RX8130, and persist the supplied display offset. Initial connect and reconnect refresh the clock; no device-side public NTP dependency or polling protocol is added. Missing/invalid time leaves the previous base unchanged. The RTC stores UTC; only presentation applies the Gateway offset. Alarm and Voice/Speech protocol behavior are unchanged. ## Agent History (0.6.3) Read-only, bounded, additive. Firmware advertises `history`; the Gateway relays it in `hello_ack` capabilities only when the device offers it. No second conversation database exists on either side: pages are normalized from native persistence on demand. Device → Gateway: ```json {"type":"history_get","request_id":"read-1","expected_target_generation":7, "target":"agent:codex:s","agent":"codex","session_id":"s","cursor":"aDoxOjQ"} ``` `cursor` is opaque and omitted for the newest page. The request is valid only for the current device connection, target, target generation, agent and session_id. Gateway → Device: ```json {"type":"history_page","request_id":"read-1","agent":"codex","generation":7, "messages":[{"role":"user","text":"..."},{"role":"assistant","text":"..."}], "next_cursor":"aDoxOjQ","oldest":false,"target":"agent:codex:s","session_id":"s"} {"type":"history_error","request_id":"read-1","message":"Stale history target"} ``` Each page holds at most four messages, is newest first, and the whole frame stays within 4096 UTF-8 bytes; each message is at most 1200 UTF-8 bytes and is truncated on a code point boundary. Only `role` `user`/`assistant` text is ever emitted: reasoning, commentary, tool output, command output, file changes, credentials and internal metadata are excluded and credentials are redacted before truncation. `next_cursor` empty with `oldest: true` means the oldest message is reached. A target switch, generation change or disconnect cancels or ignores an in-flight page; a stale page is never applied to a new session. The device caches at most three pages.