# Device Lab protocol 0.5.0

设备通过 `wss://gw.pangtiao.men/device/ws` 连接。连接后的第一帧必须是：

```json
{
  "type": "hello",
  "device_id": "esp32s3-28848543a200",
  "device_token": "<secret>",
  "board": "m5stack-stopwatch",
  "firmware": "0.5.0",
  "capabilities": ["agent_target", "approval", "voice_input", "settings", "speech_playback"]
}
```

Gateway 校验 token 和已绑定的 device id，失败时以 WebSocket policy error 关闭。成功后依次发送 `hello_ack`、最新 `server_snapshot` 和当前 `target_state`（server）或 `agent_state`（Pi/Codex）。

## Gateway → device

`server_snapshot` 每 2 秒主动推送。Gateway 只保留最新一帧以及计算网速所需的上一帧。

```json
{
  "type": "server_snapshot",
  "hostname": "lighthouse",
  "cpu_percent": 23.4,
  "load_1m": 0.42,
  "load_5m": 0.31,
  "memory_used": 4294967296,
  "memory_total": 8589934592,
  "memory_percent": 52.5,
  "swap_percent": 7.1,
  "disk_used": 987654321,
  "disk_total": 1234567890,
  "disk_percent": 80.0,
  "network_rx_bytes": 123456789,
  "network_tx_bytes": 98765432,
  "network_rx_bps": 1834212,
  "network_tx_bps": 328192,
  "uptime": 123456,
  "timestamp": "2026-10-02T12:00:00+00:00"
}
```

Server target：

```json
{"type":"target_state","target":"server"}
```

Agent target（Pi/Codex）：

```json
{
  "type": "agent_state",
  "agent": "pi",
  "target": "agent:pi:<session-id>",
  "model": "provider/model-id",
  "effort": null,
  "session_id": "<session-id>",
  "name": "esp32-dev",
  "cwd": "/home/lighthouse/esp32-dev",
  "state": "RUNNING",
  "elapsed_seconds": 151,
  "waiting_reason": null,
  "activity_type": "text",
  "latest_activity": "The target selection request...",
  "recent_text": "...recent UTF-8-safe output tail...",
  "activity_updated_at": "2026-10-02T12:00:00+00:00"
}
```

Gateway 为每个 Pi session 保留最多 12 KiB 的内存 recent history；Admin/API 默认只返回 480 B preview，设备取得 target 时返回最近 4 KiB。历史不落盘。仅选择了对应 session 的设备会收到该 session 的增量：

```json
{
  "type": "agent_activity",
  "agent": "pi",
  "target": "agent:pi:<session-id>",
  "session_id": "<session-id>",
  "activity_type": "tool",
  "activity": "Running\npytest -q",
  "updated_at": "2026-10-02T12:00:01+00:00"
}
```

真实 `RUNNING -> DONE/FAILED` 时，Gateway 只在首次 terminal transition 的 `agent_state` 中附带 `"notification":"done"` 或 `"failed"`。重复的 `agent_end/agent_settled` 不再附带通知。开发自测通过受 Cloudflare Access 管理边界保护的 `POST /devices/{device_id}/test-notification` 发送：

```json
{"kind":"done"}
```

设备 WSS 对应收到 `{"type":"notification_test","kind":"done"}`；`failed` 同理。

`display_message` 保留第一阶段格式。

## Device → Gateway

设备继续发送 `heartbeat` 和实体按键的 `button_event`。Button A/B 的本地页面切换不依赖 Gateway。

## Pi Extension → Gateway

Extension 向 loopback `POST /integrations/pi/events` 发送 `pi_session_state`，并用独立的 `X-Device-Lab-Pi-Token` header 认证。lifecycle 事件为 `session_start`、`session_info_changed`、`agent_start`、`agent_end`、`session_shutdown`。实时内容使用 `event: "activity"`，并带 `activity_type: "text" | "tool"`、`activity` 及可选 `tool_name`。Pi 端使用正式 `message_update/text_delta` 和 `tool_execution_start/end`，文本按 350 ms 或 120 字符批量发送，工具参数只保留脱敏后的短摘要；审批决定使用下述独立的 Approval control 协议与本机 Unix socket。


## Approval control (0.4.0)

After authenticated `hello`, Gateway presents the oldest still-pending approval,
regardless of the selected target. Approval does not change the stored target.

```json
{"type":"approval_request","approval_id":"opaque-unique-id","agent":"codex","session_id":"thread-id","project_id":"project-id","kind":"command","title":"Run command?","detail":"touch /tmp/proof","choices":["approve","reject"],"created_at":"2026-10-03T01:00:00Z","expires_at":null,"state":"PENDING","expires_in":null}
```

Device click/release response:

```json
{"type":"approval_response","approval_id":"opaque-unique-id","decision":"approve"}
```

B uses `reject`. This is a distinct event; `button_event` never approves anything.
Gateway requires PENDING, valid decision, and the exact ID currently presented to
this authenticated WebSocket connection. Invalid/stale/duplicate responses are
ignored and warned. Admin and device race through the same atomic resolve method.

All devices that were shown a completed request receive:

```json
{"type":"approval_cancelled","approval_id":"opaque-unique-id","state":"APPROVED"}
```

Terminal `state` is APPROVED / REJECTED / CANCELLED. EXPIRED is accepted only
for compatibility; no default path produces it. NO DECISION = KEEP WAITING. The same terminal
envelope removes a presentation on every device; APPROVED/REJECTED show one second
of feedback. EXPIRED/CANCELLED dismiss immediately. Then Gateway sends the next
FIFO pending item. Device holds only a tiny transient queue, never durable history.

Disconnect clears the device overlay and outbound decision; reconnect only replays
requests still PENDING in Gateway. Codex disconnect cancels requests from the old
App Server connection and never replays their responses on the new connection.

The unified AgentSession state includes WAITING_APPROVAL and waiting_reason. This
is a derived overlay: removing it reveals the most recent real lifecycle state.

## Voice (0.4.0)

All messages use the same authenticated `/device/ws`; no upload endpoint.

```json
{"type":"voice_start","voice_id":"random-id","sample_rate":44100,"bits":16,"channels":1}
{"type":"voice_chunk","voice_id":"random-id","sequence":0,"audio":"<base64 PCM>"}
{"type":"voice_end","voice_id":"random-id"}
{"type":"voice_cancel","voice_id":"random-id"}
{"type":"voice_decision","voice_id":"random-id","decision":"send"}
```

`voice_decision` also accepts `cancel`. Gateway pins the selected agent/session
and device connection before capture. IDs cannot be replayed on a new connection.
Gateway replies with `voice_state` (RECORDING / TRANSCRIBING / SENT / CANCELLED),
`voice_transcript` (AWAITING_CONFIRMATION, agent, session_id, name, transcript), or
`voice_error` (FAILED and a safe message). The device never sends transcript text
back: SEND refers only to the already-bound voice ID. Limits and failure semantics
are in [voice and settings](voice-input-design.md).

## Session settings (0.4.0)

```json
{"type":"settings_get","request_id":"random-id"}
{"type":"settings_apply","request_id":"random-id","target":"agent:codex:thread-id","settings":{"model":"model-from-catalog","effort":"effort-from-metadata"}}
{"type":"settings_cancel","request_id":"random-id"}
```

Gateway replies `settings_state` / `settings_updated` with the request ID, pinned
target, agent, session_id, name, model, effort, available_models, available_efforts.
Each model carries id/name/default_effort/available_efforts and optional is_default.
Quick catalog is deduplicated and capped at six. `settings_error` retains the
request ID and safe error message; Gateway values remain authoritative.
Target switch, approval and disconnect invalidate the presentation. A setting
cannot apply to a different target, and duplicate apply is ignored.

Admin uses `GET/POST /api/agents/{agent}/sessions/{id}/settings`; POST accepts only
model/effort. No approval timeout settings are exposed. Busy sessions return 409.

Legacy stored `pi:<id>` targets are interpreted as `agent:pi:<id>`. Firmware 0.4.0
also understands legacy `target_state pi:...` and `pi_activity`. Gateway emits
only the unified Agent messages; upgrade older firmware before device testing.


## Speech playback (0.5.0)

A hello without `speech_playback` remains valid, including Firmware 0.4.0. Gateway
never sends `speech_*` to such devices. Speech Mode defaults OFF; modes are stored
per device + target outside the repo. Explicit screen reading/test works in OFF.
All following device requests use the existing authenticated WSS connection.

Gateway → Device:

```json
{"type":"speech_settings","target":"agent:pi:s","mode":"OFF","request_id":null}
{"type":"speech_start","speech_id":"unique-id","target":"agent:pi:s","sample_rate":44100,"bits":16,"channels":1,"source":"reply","text":"Bounded preview."}
{"type":"speech_chunk","speech_id":"unique-id","sequence":0,"audio":"<base64 s16le PCM>"}
{"type":"speech_end","speech_id":"unique-id","chunks":1}
{"type":"speech_cancel","speech_id":"unique-id"}
{"type":"speech_error","message":"Speech unavailable"}
```

`source` is screen/reply/summary. PCM is 44100 Hz, signed 16-bit little endian mono.
Each chunk holds at most 7056 bytes (80 ms / 9408 base64 characters); sequences start
at zero. Frames stay below 10 KiB, within the 16 KiB device receive buffer. Gateway
sends at audio rate without burst catch-up. Device queues at most 8 frames; overflow
or sequence error aborts. `speech_end.chunks` must match the received count and does
not stop queued playback early. Stale IDs are ignored. Cancel clears queued PCM.

Device → Gateway:

```json
{"type":"speech_settings_get","request_id":"read-mode"}
{"type":"speech_settings_set","request_id":"set-mode","target":"agent:pi:s","mode":"MANUAL"}
{"type":"speech_request","request_id":"read-screen","target":"agent:pi:s","source":"screen","text":"Codex。任务完成。Tests passed."}
{"type":"speech_stop","speech_id":"unique-id"}
```

Setting/request identities are bounded, single-use per connection. `target` must
match the current target for mutations/screen reading, protecting against races.
Screen text is at most 1200 UTF-8 bytes. A speech_stop with no ID also cancels pending
synthesis (used when local recording/Settings/Approval preempts before speech_start).
A stop carrying an old ID cannot cancel a replacement stream. Approval and voice_start
also cancel server-side work independently. Disconnect invalidates pending work.

Admin: `POST /api/devices/{id}/speech` accepts target/mode;
`POST /api/devices/{id}/speech-test` accepts only target and speaks a fixed sentence.
Existing Cloudflare Access protection applies. Model/effort settings are unchanged.
See [Speech design](spoken-agent-design.md) for UI, provider and final-reply semantics.