Interesting -- PLAN.md line 193 says module splitting starts at Phase 1, but the user's design constraints say "Keep all changes in main.py (no module split yet per PLAN.md)." I will respect the user's explicit constraint and keep everything in `main.py`, deferring module splitting.

Now I have all the information I need. Let me produce the implementation plan.

---

## Phase 1 Implementation Plan: Audio Buffering / Press-to-Talk

### 1. Summary of Current State

**11 tests pass.** The codebase is a single `main.py` (309 lines) with a single test file `tests/test_main.py` (212 lines). There are no per-connection state objects. Every binary WebSocket frame independently triggers the full STT-to-LLM-to-TTS pipeline, which is the core problem this phase solves.

The existing frame router at line 193 (`handle_text_frame`) recognizes `KEY_DOWN`, `KEY_UP`, and `WAKE_WORD` via Pydantic validation, but only acts on `WAKE_WORD`. `KEY_DOWN` and `KEY_UP` are parsed then silently dropped. The binary frame handler at line 219 (`handle_binary_frame`) runs the entire pipeline on each frame.

---

### 2. Design Decisions

**Q1: Where to store SessionState?**
Decision: **Local variable in `websocket_chat`**, passed by reference to all handler functions. This is the simplest approach -- no module-level dict to worry about cleanup, no risk of stale entries after disconnect, and no concurrency issues since each WebSocket connection is a single asyncio task. The `SessionState` instance is created at the top of `websocket_chat` and passed to `handle_text_frame` and `handle_binary_frame`.

**Q2: wake_word vs key_down?**
Decision: Both trigger the `idle -> listening` transition identically. The state machine treats them as synonyms for "start recording."

**Q3: Max buffer size?**
Decision: **10 MB** (`MAX_AUDIO_BUFFER_BYTES = 10 * 1024 * 1024`). At 24kbps Opus, that is roughly 55 minutes of audio, which is generous. If the buffer exceeds this limit during `listening`, log a warning and auto-trigger the pipeline (treat it as an implicit key_up).

**Q4: Empty audio on key_up?**
Decision: Log a warning (`"key_up received but audio buffer is empty"`) and transition directly back to `idle`. Do not invoke the pipeline.

**Q5: Binary frames while idle?**
Decision: **Drop with a warning log** (strict state machine). The ESP32 firmware is expected to send `key_down` first. This prevents accidental pipeline invocations.

**Q6: Binary frames during processing?**
Decision: **Drop with a warning log.** Cannot start a new recording while the pipeline is running. The client must wait for the pipeline to finish (state returns to `idle`).

---

### 3. Step-by-Step Implementation

#### Step 0: Write failing tests first (TDD, per CLAUDE.md)

Add the following new test functions to `tests/test_main.py`. Write ALL failing tests before touching `main.py`. New tests:

1. **`test_session_state_defaults`** -- Unit test. Instantiate `SessionState()`, verify `status == "idle"` and `audio_chunks == []`.

2. **`test_session_state_validation`** -- Unit test. Verify `SessionState(status="invalid")` raises `ValidationError`.

3. **`test_key_down_starts_listening`** -- Integration test. Send `key_down` over WebSocket, verify response is `{"type": "state", "status": "listening"}`.

4. **`test_wake_word_starts_listening`** -- This already exists as `test_websocket_text_frame_wake_word`. No new test needed, but verify it still passes after refactor.

5. **`test_key_up_triggers_pipeline`** -- Integration test. Send `key_down`, then binary frames, then `key_up`. Mock `process_stt`, `call_openclaw`, `process_tts`. Verify the full response sequence: `listening` state, `thinking` state, `text_reply`, binary audio. Verify `process_stt` is called with the concatenated audio.

6. **`test_key_up_empty_buffer_skips_pipeline`** -- Integration test. Send `key_down` then immediately `key_up` with no binary frames between. Verify NO `thinking` state or pipeline output. Verify state returns to `idle` (no crash, connection stays open -- send a subsequent `wake_word` to verify connectivity).

7. **`test_binary_frame_while_idle_is_dropped`** -- Integration test. Send binary data without prior `key_down`. Verify it does not trigger the pipeline (send `wake_word` after, verify `listening` response to confirm connection is alive).

8. **`test_binary_frames_are_concatenated`** -- Integration test. Send `key_down`, then 3 separate binary frames, then `key_up`. Mock `process_stt` and verify it was called with all 3 frames joined: `b"chunk1" + b"chunk2" + b"chunk3"`.

9. **`test_repeated_key_down_resets_buffer`** -- Integration test. Send `key_down`, binary data, then another `key_down`. The second `key_down` should clear the buffer and restart. Then send `key_up`. Verify `process_stt` is called only with audio received after the second `key_down`.

10. **`test_key_up_without_key_down_is_ignored`** -- Integration test. Send `key_up` while in `idle` state. Verify no crash, no pipeline invocation. Verify connection stays alive.

11. **`test_oversized_buffer_auto_triggers_pipeline`** -- Unit test on `handle_binary_frame`. Verify that when `audio_chunks` total exceeds `MAX_AUDIO_BUFFER_BYTES`, the handler auto-triggers the pipeline. (This one may be deferred as an enhancement -- see section 5.)

Total: 8-9 new test functions (reaching approximately 19-20 tests total).

**Important existing test to update:** `test_websocket_binary_frame_full_pipeline` (line 171). Currently it sends a bare binary frame and expects the full pipeline response. After Phase 1, a bare binary frame in `idle` state will be dropped. This test must be rewritten to follow the `key_down -> binary -> key_up` sequence. This is the only breaking change to existing tests.

#### Step 1: Add `SessionState` model and `MAX_AUDIO_BUFFER_BYTES` constant

Location: `/root/projects/openclaw-proxy/main.py`, after the existing Pydantic models (after line 64).

```python
# ── Audio buffer configuration ──────────────────────────────────────
MAX_AUDIO_BUFFER_BYTES: int = 10 * 1024 * 1024  # 10 MB

class SessionState(BaseModel):
    """Per-connection state for audio buffering.

    Attributes:
        status: Current state in the press-to-talk state machine.
        audio_chunks: Buffered audio frames received during "listening".
    """

    model_config = {"arbitrary_types_allowed": True}

    status: Literal["idle", "listening", "processing"] = "idle"
    audio_chunks: list[bytes] = []

    def buffer_size(self) -> int:
        """Return total size of buffered audio in bytes."""
        return sum(len(chunk) for chunk in self.audio_chunks)

    def drain_audio(self) -> bytes:
        """Join and clear all buffered audio. Return concatenated bytes."""
        audio = b"".join(self.audio_chunks)
        self.audio_chunks.clear()
        return audio

    def reset(self) -> None:
        """Reset to idle state with empty buffer."""
        self.status = "idle"
        self.audio_chunks.clear()
```

The `Literal` import must be added to the top-level imports: `from typing import Literal`.

Note: `drain_audio` and `reset` are helper methods. They keep the state transitions centralized and testable. `buffer_size` is used for the overflow check.

#### Step 2: Refactor `handle_text_frame` to manage state transitions

Change the function signature to accept `SessionState`:

```python
async def handle_text_frame(
    ws: WebSocket, raw: str, state: SessionState, session_id: str
) -> None:
```

Add logic for each action:

- **`WAKE_WORD`**: If `state.status == "idle"`, set `state.status = "listening"`, clear `state.audio_chunks`, send `{"type": "state", "status": "listening"}`. If already `listening`, log a warning and re-send the acknowledgement (idempotent). If `processing`, log a warning and ignore.

- **`KEY_DOWN`**: Identical behavior to `WAKE_WORD`. If `idle`, transition to `listening`. If already `listening`, log warning about repeated `key_down`, clear the buffer (the user re-pressed), re-send `listening` state. If `processing`, log warning and ignore.

- **`KEY_UP`**: If `state.status == "listening"`, transition to `processing` and call `handle_pipeline`. If `idle`, log warning ("key_up without prior key_down") and ignore. If `processing`, log warning and ignore.

The `session_id` parameter is needed because `handle_text_frame` now triggers the pipeline on `KEY_UP`.

#### Step 3: Refactor `handle_binary_frame` to buffer-only

Change the function signature:

```python
async def handle_binary_frame(
    ws: WebSocket, data: bytes, state: SessionState
) -> None:
```

New logic (replaces the entire current body):

1. If `state.status != "listening"`: log warning ("binary frame received in {state.status} state, dropping") and return.
2. Check `state.buffer_size() + len(data) > MAX_AUDIO_BUFFER_BYTES`. If exceeded, log warning and return (drop the frame rather than auto-triggering pipeline -- keep it simple for Phase 1).
3. Append: `state.audio_chunks.append(data)`.
4. Log info: `"Buffered %d bytes (total: %d bytes)"`.

This function no longer calls `process_stt`, `call_openclaw`, or `process_tts`. It only appends to the buffer.

#### Step 4: Extract `handle_pipeline` from the old `handle_binary_frame`

New function containing the STT-to-LLM-to-TTS logic currently in lines 232-261:

```python
async def handle_pipeline(
    ws: WebSocket, state: SessionState, session_id: str
) -> None:
    """Run the full STT -> LLM -> TTS pipeline on buffered audio.

    Args:
        ws: Active WebSocket connection.
        state: Per-connection session state (must be in "processing" status).
        session_id: Per-connection session ID.

    The state is reset to "idle" after the pipeline completes (or on error).
    """
```

Body:
1. Assert / guard: `state.status == "processing"`.
2. `audio = state.drain_audio()`.
3. If `len(audio) == 0`: log warning, `state.reset()`, return early.
4. `try:` block containing the existing pipeline stages (STT, thinking state, heartbeat, LLM, text_reply, TTS, binary send).
5. `finally:` block: `state.reset()` -- ensures state always returns to `idle`, even on error.

The error handling in the `finally` block is important: if `process_stt` or `call_openclaw` raises an exception, the connection must not remain stuck in `processing` forever. The `finally` guarantees the state resets to `idle` so the user can try again.

#### Step 5: Update `websocket_chat` to create and pass `SessionState`

Modify the WebSocket endpoint function:

```python
@app.websocket("/chat")
async def websocket_chat(ws: WebSocket) -> None:
    """Main WebSocket endpoint for XiaoZhi ESP32 hardware."""
    await ws.accept()
    peer = f"{ws.client.host}:{ws.client.port}" if ws.client else "unknown"
    session_id = f"xiaozhi-{uuid.uuid4().hex[:8]}"
    state = SessionState()
    logger.info("Client connected: %s (session=%s)", peer, session_id)

    try:
        while True:
            message = await ws.receive()

            if message["type"] == "websocket.disconnect":
                break

            text_data = message.get("text")
            byte_data = message.get("bytes")

            if text_data is not None:
                await handle_text_frame(ws, text_data, state, session_id)
            elif byte_data is not None:
                await handle_binary_frame(ws, byte_data, state)
    except WebSocketDisconnect:
        pass
    finally:
        logger.info("Client disconnected: %s (session=%s)", peer, session_id)
```

Changes: add `state = SessionState()` after `session_id`, pass `state` and `session_id` to both handlers.

#### Step 6: Update existing test `test_websocket_binary_frame_full_pipeline`

This is the ONE existing test that must change. Currently it sends a raw binary frame and expects the pipeline response. After Phase 1, raw binary frames in `idle` state are dropped.

Rewrite to follow the press-to-talk sequence:

```python
def test_websocket_binary_frame_full_pipeline() -> None:
    """Press-to-talk: key_down -> binary -> key_up triggers STT -> LLM -> TTS."""
    client = TestClient(app)
    with (
        patch("main.process_stt", new_callable=AsyncMock, return_value="..."),
        patch("main.call_openclaw", new_callable=AsyncMock, return_value="..."),
        patch("main.process_tts", new_callable=AsyncMock, return_value=b"..."),
    ):
        with client.websocket_connect("/chat") as ws:
            # 1) key_down -> listening
            ws.send_text(json.dumps({"type": "hardware_event", "action": "key_down"}))
            resp = json.loads(ws.receive_text())
            assert resp == {"type": "state", "status": "listening"}

            # 2) Binary audio frames
            ws.send_bytes(b"\x00" * 80)

            # 3) key_up -> triggers pipeline
            ws.send_text(json.dumps({"type": "hardware_event", "action": "key_up"}))

            # 4) Expect: thinking, text_reply, binary
            msg1 = json.loads(ws.receive_text())
            assert msg1 == {"type": "state", "status": "thinking"}

            msg2 = json.loads(ws.receive_text())
            assert msg2["type"] == "text_reply"

            audio = ws.receive_bytes()
            assert isinstance(audio, bytes)
```

#### Step 7: Run all tests and verify green

Expected: all 11 existing tests pass (with the one rewrite), plus 8-9 new tests pass. Total approximately 19-20 tests.

---

### 4. Function Signature Changes Summary

| Function | Current Signature | New Signature |
|----------|------------------|---------------|
| `handle_text_frame` | `(ws, raw)` | `(ws, raw, state, session_id)` |
| `handle_binary_frame` | `(ws, data, session_id)` | `(ws, data, state)` |
| `handle_pipeline` | (does not exist) | `(ws, state, session_id)` -- NEW |
| `websocket_chat` | unchanged | unchanged (signature stays same, creates `SessionState` internally) |
| `process_stt` | unchanged | unchanged |
| `call_openclaw` | unchanged | unchanged |
| `process_tts` | unchanged | unchanged |

---

### 5. Edge Cases and Guard Rails

| Scenario | Behavior |
|----------|----------|
| Binary frame while `idle` | Drop, log warning |
| Binary frame while `processing` | Drop, log warning |
| `key_up` while `idle` | Ignore, log warning |
| `key_up` while `processing` | Ignore, log warning |
| `key_down` while `listening` (repeated) | Log warning, clear buffer, re-send `listening` ack |
| `key_down` while `processing` | Ignore, log warning |
| Empty audio on `key_up` | Skip pipeline, reset to `idle`, log warning |
| Buffer exceeds 10MB | Drop incoming frame, log warning |
| Pipeline raises exception | `finally` block resets state to `idle` |
| WebSocket disconnects mid-pipeline | `WebSocketDisconnect` exception propagates normally; `SessionState` is garbage collected with the local variable |

---

### 6. What Does NOT Change

- `process_stt`, `call_openclaw`, `process_tts` -- internals untouched.
- `HardwareAction`, `HardwareEvent`, `StateNotification`, `TextReply` -- no changes.
- `/health` endpoint -- no changes.
- WebSocket protocol messages -- no new message types added.
- `_get_whisper_model` -- no changes.
- Configuration constants -- no changes (one new constant `MAX_AUDIO_BUFFER_BYTES` added).
- 6 pipeline unit tests (`test_process_stt_*`, `test_call_openclaw_*`, `test_process_tts_*`) -- completely untouched.
- 3 integration tests (`test_health_endpoint`, `test_websocket_connect_and_disconnect`, `test_websocket_text_frame_malformed_json`) -- untouched.
- `test_websocket_text_frame_wake_word` -- should continue passing because `wake_word` still sends `listening` response.

---

### 7. Sequencing and Dependencies

The implementation order matters because of TDD:

1. **Write new tests first** (they will fail because `SessionState` does not exist and handler signatures have not changed).
2. **Add `SessionState` model and constant** to `main.py` -- makes `test_session_state_defaults` and `test_session_state_validation` pass.
3. **Refactor `handle_text_frame`** -- makes `test_key_down_starts_listening`, `test_key_up_without_key_down_is_ignored`, and `test_repeated_key_down_resets_buffer` pass.
4. **Refactor `handle_binary_frame`** -- makes `test_binary_frame_while_idle_is_dropped` pass.
5. **Extract `handle_pipeline`** + wire up `KEY_UP` in `handle_text_frame` -- makes `test_key_up_triggers_pipeline`, `test_key_up_empty_buffer_skips_pipeline`, and `test_binary_frames_are_concatenated` pass.
6. **Update `websocket_chat`** to instantiate `SessionState` and pass it to handlers.
7. **Rewrite `test_websocket_binary_frame_full_pipeline`** -- fixes the one breaking existing test.
8. **Run full test suite** -- verify all 19-20 tests pass.

Practically, steps 3-6 are tightly coupled and will be done together, but the test-writing happens first.

---

### 8. Risks

| Risk | Mitigation |
|------|-----------|
| Starlette `TestClient` synchronous WebSocket does not support interleaved send/receive well for complex sequences | Keep test sequences simple: send-all-then-receive-all within each test. The pipeline is invoked synchronously within the event loop, so responses are queued before `receive` calls. |
| `handle_pipeline` is called from within `handle_text_frame` (on `KEY_UP`). This means the WebSocket read loop is blocked during pipeline execution. | This is the existing behavior (the current `handle_binary_frame` already blocks the loop). This is acceptable for Phase 1 single-user. Phase 5 should consider `asyncio.create_task` for the pipeline. |
| `Literal` import may be missing | Add `from typing import Literal` at the top. Python 3.11 supports this natively. |
| The existing `test_websocket_text_frame_wake_word` test does not check state transitions | It only checks that `listening` is sent back. The test still passes because `wake_word` still sends `listening`. No change needed. |

---

### 9. Deferred Items (Explicitly NOT in Phase 1)

- **Module splitting** (PLAN.md mentions `session.py`, `models.py`) -- deferred per user constraint.
- **Opus frame chunking for TTS output** -- listed as Phase 5 in PLAN.md.
- **`speaking` state notification** -- listed as Phase 5 in PLAN.md.
- **Auto-trigger pipeline on buffer overflow** -- keep it simple, just drop frames. Can add auto-trigger later.
- **Concurrent pipeline execution** (`asyncio.create_task`) -- Phase 5 production hardening.

---

### Critical Files for Implementation

- `/root/projects/openclaw-proxy/main.py` - Core file to modify: add SessionState model, refactor handle_text_frame/handle_binary_frame, extract handle_pipeline, update websocket_chat
- `/root/projects/openclaw-proxy/tests/test_main.py` - Write 8-9 new test functions, rewrite 1 existing test (test_websocket_binary_frame_full_pipeline)
- `/root/projects/openclaw-proxy/PLAN.md` - Update Phase 1 checkboxes after implementation is verified
- `/root/projects/openclaw-proxy/test_client.py` - Update manual test client to follow key_down/binary/key_up sequence instead of bare binary frame

agentId: a614ac6e5ef9fbcdc (for resuming to continue this agent's work if needed)
<usage>total_tokens: 36371
tool_uses: 15
duration_ms: 173700</usage>

Prompt:
Design the implementation plan for Phase 1 (Audio Buffering / press-to-talk) of the openclaw-proxy project.

## Current Architecture

**main.py** (309 lines) has these key components:

### Pydantic Models (lines 34-64):
- `HardwareAction(str, Enum)`: KEY_DOWN, KEY_UP, WAKE_WORD
- `HardwareEvent(BaseModel)`: type + action
- `StateNotification(BaseModel)`: type="state" + status
- `TextReply(BaseModel)`: type="text_reply" + text

### Frame Routing:
- `handle_text_frame(ws, raw)` (lines 193-216): Parses JSON, validates HardwareEvent. Only handles WAKE_WORD (sends "listening" state). KEY_DOWN and KEY_UP are parsed but **ignored**.
- `handle_binary_frame(ws, data, session_id)` (lines 219-261): Each binary frame **independently** triggers the full STT→LLM→TTS pipeline. This is the core problem.

### WebSocket endpoint (lines 269-294):
```python
async def websocket_chat(ws: WebSocket) -> None:
    await ws.accept()
    session_id = f"xiaozhi-{uuid.uuid4().hex[:8]}"
    try:
        while True:
            message = await ws.receive()
            if message["type"] == "websocket.disconnect": break
            text_data = message.get("text")
            byte_data = message.get("bytes")
            if text_data is not None:
                await handle_text_frame(ws, text_data)
            elif byte_data is not None:
                await handle_binary_frame(ws, byte_data, session_id)
    except WebSocketDisconnect: pass
```

No per-connection state exists beyond `session_id`.

## Phase 1 Requirements (from PLAN.md)

1. Implement `SessionState` per connection:
```python
class SessionState(BaseModel):
    status: Literal["idle", "listening", "processing"] = "idle"
    audio_chunks: list[bytes] = []
```

2. State machine:
```
idle ─(key_down/wake_word)─> listening ─(key_up)─> processing ─(done)─> idle
```

3. Binary frames in "listening" state: append to audio_chunks buffer
4. key_up: join all audio_chunks, pass complete audio to pipeline
5. Refactor handle_binary_frame to just append
6. New handle_pipeline(ws, session_state, session_id) for unified STT→LLM→TTS
7. Edge cases: empty audio, oversized audio (need max buffer), repeated key_down, key_up without key_down, binary while idle

## Design Constraints

- CLAUDE.md rules: TDD, type hints mandatory, Pydantic models, Google-style docstrings
- Keep all changes in main.py (no module split yet per PLAN.md)
- Don't modify process_stt, call_openclaw, process_tts internals
- Don't modify the WebSocket protocol or health endpoint
- Existing 11 tests must continue passing (may need mock updates)

## Questions to Address in the Plan

1. Where to store SessionState? Options:
   a. Local variable in websocket_chat (simplest, passed to handlers)
   b. Module-level dict keyed by ws or session_id

2. How to handle wake_word vs key_down? Both should start listening. PLAN.md says both trigger buffering.

3. What max buffer size? Suggest 10MB or 60 seconds of audio.

4. What to do with empty audio on key_up? Log warning and skip pipeline.

5. What to do with binary frames while in "idle" state? Options:
   a. Drop silently (strict state machine)
   b. Auto-enter listening and buffer (lenient)

   I recommend option (a) - drop with a warning log. The ESP32 firmware should send key_down first.

6. How to handle binary frames during "processing" state? Drop them - can't start new recording while pipeline is running.

7. Test strategy: What new tests are needed?

Please provide a detailed step-by-step implementation plan with code changes needed for each file.


Excellent! Now I have a thorough understanding and a detailed plan from the Plan agent. Let me now read the existing plan file and write the final plan.


Now let me write the final plan file.


     1→# Plan: Real STT + TTS Integration
     2→
     3→## Context
     4→
     5→openclaw-proxy Phase 0.5 已跑通：ESP32 ↔ WebSocket ↔ OpenClaw CLI 对话链路完整，但 STT 和 TTS 仍是 mock。
     6→现在需要接入真实的 Whisper（STT）和 edge-tts（TTS），让语音输入→文字→语音输出全链路打通。
     7→
     8→**环境确认（均已安装）：**
     9→- openai-whisper 20250625 + PyTorch 2.10.0 (CPU)
    10→- edge-tts 7.2.7
    11→- ffmpeg 7.0.2 (libopus encoder/decoder)
    12→- 4GB 可用内存，4 核 CPU，无 GPU
    13→
    14→## 实现方案
    15→
    16→### 1. STT — `process_stt(audio_bytes) -> str`
    17→
    18→**链路：** Opus bytes → ffmpeg → 16kHz WAV → Whisper base 模型 → 文本
    19→
    20→**实现：**
    21→- 模块级加载 Whisper base 模型（一次加载，全局复用）
    22→- 用 `asyncio.to_thread()` 包装 Whisper 推理（CPU 密集，不能阻塞事件循环）
    23→- Whisper 的 `load_audio()` 内部会调 ffmpeg 处理任意格式，所以只需把 Opus 写入临时文件传路径即可
    24→- 用 `tempfile.NamedTemporaryFile` 写 Opus 字节，传路径给 `model.transcribe()`
    25→
    26→**关键代码骨架：**
    27→```python
    28→import whisper
    29→_whisper_model = whisper.load_model("base")
    30→
    31→async def process_stt(audio_bytes: bytes) -> str:
    32→    with tempfile.NamedTemporaryFile(suffix=".opus", delete=True) as f:
    33→        f.write(audio_bytes)
    34→        f.flush()
    35→        result = await asyncio.to_thread(
    36→            _whisper_model.transcribe, f.name, language="zh"
    37→        )
    38→    return result["text"].strip()
    39→```
    40→
    41→**修改文件：** `main.py` 第 54-86 行（`process_stt` 函数）+ 模块顶部导入
    42→
    43→### 2. TTS — `process_tts(text) -> bytes`
    44→
    45→**链路：** 文本 → edge-tts (MP3) → ffmpeg → Opus bytes
    46→
    47→**实现：**
    48→- edge-tts 是异步的，用 `Communicate(text, voice).stream()` 获取 MP3 流
    49→- 收集完整 MP3 字节后，用 ffmpeg subprocess 转 Opus：
    50→  `ffmpeg -i pipe:0 -c:a libopus -b:a 24k -ar 16000 -ac 1 -f opus pipe:1`
    51→- 返回完整 Opus 字节
    52→
    53→**关键代码骨架：**
    54→```python
    55→import edge_tts
    56→EDGE_TTS_VOICE = "zh-CN-XiaoxiaoNeural"
    57→
    58→async def process_tts(text: str) -> bytes:
    59→    # 1. edge-tts → MP3
    60→    communicate = edge_tts.Communicate(text=text, voice=EDGE_TTS_VOICE)
    61→    mp3_chunks = []
    62→    async for chunk in communicate.stream():
    63→        if chunk["type"] == "audio":
    64→            mp3_chunks.append(chunk["data"])
    65→    mp3_bytes = b"".join(mp3_chunks)
    66→
    67→    # 2. MP3 → Opus via ffmpeg
    68→    proc = await asyncio.create_subprocess_exec(
    69→        "ffmpeg", "-i", "pipe:0", "-c:a", "libopus",
    70→        "-b:a", "24k", "-ar", "16000", "-ac", "1", "-f", "opus", "pipe:1",
    71→        stdin=asyncio.subprocess.PIPE,
    72→        stdout=asyncio.subprocess.PIPE,
    73→        stderr=asyncio.subprocess.PIPE,
    74→    )
    75→    opus_bytes, _ = await proc.communicate(input=mp3_bytes)
    76→    return opus_bytes
    77→```
    78→
    79→**修改文件：** `main.py` 第 100-122 行（`process_tts` 函数）+ 模块顶部导入
    80→
    81→### 3. 配置常量
    82→
    83→在 `main.py` 顶部 Configuration 区域新增：
    84→```python
    85→WHISPER_MODEL_SIZE = "base"
    86→EDGE_TTS_VOICE = "zh-CN-XiaoxiaoNeural"
    87→FFMPEG_BIN = which("ffmpeg") or "ffmpeg"
    88→```
    89→
    90→### 4. requirements.txt 更新
    91→
    92→新增：
    93→```
    94→openai-whisper>=20250625
    95→edge-tts>=7.0
    96→```
    97→（系统已安装，写入 requirements.txt 声明依赖）
    98→
    99→### 5. 测试更新
   100→
   101→- `test_process_stt` — mock `whisper.load_model` 和 `model.transcribe`，验证 tempfile 和结果解析
   102→- `test_process_tts` — mock `edge_tts.Communicate` 和 ffmpeg subprocess，验证 MP3→Opus 管线
   103→- 保持现有 10 个测试不变（mock subprocess 的 call_openclaw 测试不受影响）
   104→
   105→## 不改的东西
   106→
   107→- `.openclaw/` 目录 — 不动
   108→- `call_openclaw()` — 已经是真实 CLI 调用，不动
   109→- WebSocket 协议 / 帧路由 / 心跳逻辑 — 不动
   110→
   111→## 验证方式
   112→
   113→```bash
   114→# 1. 跑单元测试
   115→source venv/bin/activate && python -m pytest tests/ -v
   116→
   117→# 2. 启动服务
   118→python main.py
   119→
   120→# 3. 用 test_client.py 测试完整链路（mock Opus → real STT → real LLM → real TTS → Opus）
   121→python test_client.py
   122→```
   123→

# Plan: Real STT + TTS Integration

## Context

openclaw-proxy Phase 0.5 已跑通：ESP32 ↔ WebSocket ↔ OpenClaw CLI 对话链路完整，但 STT 和 TTS 仍是 mock。
现在需要接入真实的 Whisper（STT）和 edge-tts（TTS），让语音输入→文字→语音输出全链路打通。

**环境确认（均已安装）：**
- openai-whisper 20250625 + PyTorch 2.10.0 (CPU)
- edge-tts 7.2.7
- ffmpeg 7.0.2 (libopus encoder/decoder)
- 4GB 可用内存，4 核 CPU，无 GPU

## 实现方案

### 1. STT — `process_stt(audio_bytes) -> str`

**链路：** Opus bytes → ffmpeg → 16kHz WAV → Whisper base 模型 → 文本

**实现：**
- 模块级加载 Whisper base 模型（一次加载，全局复用）
- 用 `asyncio.to_thread()` 包装 Whisper 推理（CPU 密集，不能阻塞事件循环）
- Whisper 的 `load_audio()` 内部会调 ffmpeg 处理任意格式，所以只需把 Opus 写入临时文件传路径即可
- 用 `tempfile.NamedTemporaryFile` 写 Opus 字节，传路径给 `model.transcribe()`

**关键代码骨架：**
```python
import whisper
_whisper_model = whisper.load_model("base")

async def process_stt(audio_bytes: bytes) -> str:
    with tempfile.NamedTemporaryFile(suffix=".opus", delete=True) as f:
        f.write(audio_bytes)
        f.flush()
        result = await asyncio.to_thread(
            _whisper_model.transcribe, f.name, language="zh"
        )
    return result["text"].strip()
```

**修改文件：** `main.py` 第 54-86 行（`process_stt` 函数）+ 模块顶部导入

### 2. TTS — `process_tts(text) -> bytes`

**链路：** 文本 → edge-tts (MP3) → ffmpeg → Opus bytes

**实现：**
- edge-tts 是异步的，用 `Communicate(text, voice).stream()` 获取 MP3 流
- 收集完整 MP3 字节后，用 ffmpeg subprocess 转 Opus：
  `ffmpeg -i pipe:0 -c:a libopus -b:a 24k -ar 16000 -ac 1 -f opus pipe:1`
- 返回完整 Opus 字节

**关键代码骨架：**
```python
import edge_tts
EDGE_TTS_VOICE = "zh-CN-XiaoxiaoNeural"

async def process_tts(text: str) -> bytes:
    # 1. edge-tts → MP3
    communicate = edge_tts.Communicate(text=text, voice=EDGE_TTS_VOICE)
    mp3_chunks = []
    async for chunk in communicate.stream():
        if chunk["type"] == "audio":
            mp3_chunks.append(chunk["data"])
    mp3_bytes = b"".join(mp3_chunks)

    # 2. MP3 → Opus via ffmpeg
    proc = await asyncio.create_subprocess_exec(
        "ffmpeg", "-i", "pipe:0", "-c:a", "libopus",
        "-b:a", "24k", "-ar", "16000", "-ac", "1", "-f", "opus", "pipe:1",
        stdin=asyncio.subprocess.PIPE,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    opus_bytes, _ = await proc.communicate(input=mp3_bytes)
    return opus_bytes
```

**修改文件：** `main.py` 第 100-122 行（`process_tts` 函数）+ 模块顶部导入

### 3. 配置常量

在 `main.py` 顶部 Configuration 区域新增：
```python
WHISPER_MODEL_SIZE = "base"
EDGE_TTS_VOICE = "zh-CN-XiaoxiaoNeural"
FFMPEG_BIN = which("ffmpeg") or "ffmpeg"
```

### 4. requirements.txt 更新

新增：
```
openai-whisper>=20250625
edge-tts>=7.0
```
（系统已安装，写入 requirements.txt 声明依赖）

### 5. 测试更新

- `test_process_stt` — mock `whisper.load_model` 和 `model.transcribe`，验证 tempfile 和结果解析
- `test_process_tts` — mock `edge_tts.Communicate` 和 ffmpeg subprocess，验证 MP3→Opus 管线
- 保持现有 10 个测试不变（mock subprocess 的 call_openclaw 测试不受影响）

## 不改的东西

- `.openclaw/` 目录 — 不动
- `call_openclaw()` — 已经是真实 CLI 调用，不动
- WebSocket 协议 / 帧路由 / 心跳逻辑 — 不动

## 验证方式

```bash
# 1. 跑单元测试
source venv/bin/activate && python -m pytest tests/ -v

# 2. 启动服务
python main.py

# 3. 用 test_client.py 测试完整链路（mock Opus → real STT → real LLM → real TTS → Opus）
python test_client.py
```


Write

# Plan: Phase 1 — Audio Buffering (Press-to-Talk)

## Context

Phase 0/0.5/2/4 已完成：WebSocket 握手、OpenClaw CLI 对接、真实 STT (Whisper) 和 TTS (edge-tts) 均已跑通，11 个测试全部通过。

**当前问题：** 每个二进制帧独立触发完整 STT→LLM→TTS 管线。ESP32 按住按键说话时会发送多个 Opus 音频帧，当前每帧分别识别和回复，导致乱序错误。

**目标：** 实现 press-to-talk 状态机——`key_down`/`wake_word` 开始缓冲，二进制帧追加到缓冲区，`key_up` 合并音频后触发一次管线。

## 设计决策

| 问题 | 决策 |
|------|------|
| SessionState 存储位置 | `websocket_chat` 局部变量，传引用给 handler（最简单，无需清理） |
| wake_word vs key_down | 行为一致，均触发 idle→listening 转换 |
| 最大缓冲 | 10MB（`MAX_AUDIO_BUFFER_BYTES`），超限 drop 帧并 log warning |
| 空音频 key_up | Log warning，跳过管线，直接回 idle |
| idle 状态收到二进制帧 | Drop + log warning（严格状态机） |
| processing 状态收到二进制帧 | Drop + log warning |
| 管线异常 | `finally` 块保证 state 回 idle |

## 状态机

```
idle ──(key_down/wake_word)──► listening ──(key_up)──► processing ──(done/error)──► idle
         send "listening"       append audio            drain → STT→LLM→TTS
```

## 修改文件

### 1. `tests/test_main.py` — 先写失败测试（TDD）

**新增测试（8 个）：**

| 测试 | 类型 | 验证点 |
|------|------|--------|
| `test_session_state_defaults` | 单元 | status=="idle", audio_chunks==[] |
| `test_key_down_starts_listening` | 集成 | key_down → 返回 listening 状态 |
| `test_key_up_triggers_pipeline` | 集成 | key_down → binary → key_up → thinking + text_reply + audio |
| `test_key_up_empty_buffer_skips_pipeline` | 集成 | key_down → key_up（无音频）→ 不触发管线，连接存活 |
| `test_binary_frame_while_idle_is_dropped` | 集成 | idle 状态发 binary → 不触发管线，连接存活 |
| `test_binary_frames_are_concatenated` | 集成 | 3 个 binary 帧合并后传给 process_stt |
| `test_repeated_key_down_resets_buffer` | 集成 | key_down → binary → key_down → binary → key_up → 只有第二次的音频 |
| `test_key_up_without_key_down_is_ignored` | 集成 | idle 状态发 key_up → 不崩溃，连接存活 |

**修改 1 个现有测试：**
- `test_websocket_binary_frame_full_pipeline` → 改为 key_down → binary → key_up 序列

### 2. `main.py` — 实现变更

#### 2a. 新增导入

```python
from typing import Literal  # 第 1-14 行区域
```

#### 2b. 新增常量（Configuration 区域）

```python
MAX_AUDIO_BUFFER_BYTES: int = 10 * 1024 * 1024  # 10 MB
```

#### 2c. 新增 `SessionState` 模型（Pydantic models 区域后，约第 65 行后）

```python
class SessionState(BaseModel):
    status: Literal["idle", "listening", "processing"] = "idle"
    audio_chunks: list[bytes] = []

    def buffer_size(self) -> int:
        return sum(len(c) for c in self.audio_chunks)

    def drain_audio(self) -> bytes:
        audio = b"".join(self.audio_chunks)
        self.audio_chunks.clear()
        return audio

    def reset(self) -> None:
        self.status = "idle"
        self.audio_chunks.clear()
```

#### 2d. 重构 `handle_text_frame`

签名变更：`(ws, raw)` → `(ws, raw, state, session_id)`

新增逻辑：
- `KEY_DOWN` / `WAKE_WORD`：idle→listening（send "listening"）；listening 时 reset buffer 重新开始
- `KEY_UP`：listening→processing，调用 `handle_pipeline`；idle 时 ignore

#### 2e. 重构 `handle_binary_frame`

签名变更：`(ws, data, session_id)` → `(ws, data, state)`

新逻辑：仅在 listening 状态 append，其余状态 drop。检查 buffer 大小上限。
**不再调用 process_stt/call_openclaw/process_tts。**

#### 2f. 新增 `handle_pipeline(ws, state, session_id)`

从旧 `handle_binary_frame` 提取 STT→LLM→TTS 逻辑：
1. `audio = state.drain_audio()`
2. 空音频 → log + reset + return
3. `try:` 执行 STT → thinking → heartbeat + LLM → text_reply → TTS → binary
4. `finally: state.reset()` 保证回 idle

#### 2g. 更新 `websocket_chat`

```python
state = SessionState()  # 新增
# 传 state 给 handler：
await handle_text_frame(ws, text_data, state, session_id)
await handle_binary_frame(ws, byte_data, state)
```

### 3. `test_client.py` — 更新手动测试脚本

改为 key_down → binary → key_up 序列。

## 不变的部分

- `process_stt`, `call_openclaw`, `process_tts` — 内部不变
- Pydantic 消息模型（HardwareAction/Event, StateNotification, TextReply）— 不变
- `/health` 端点 — 不变
- 6 个管线单元测试 — 不变
- requirements.txt — 不变

## 验证

```bash
source venv/bin/activate && python -m pytest tests/ -v
# 预期：~19 个测试全部通过
```


The file /root/.claude/plans/fuzzy-wandering-puzzle.md has been updated successfully.

# Plan: Phase 1 — Audio Buffering (Press-to-Talk)

## Context

Phase 0/0.5/2/4 已完成：WebSocket 握手、OpenClaw CLI 对接、真实 STT (Whisper) 和 TTS (edge-tts) 均已跑通，11 个测试全部通过。

**当前问题：** 每个二进制帧独立触发完整 STT→LLM→TTS 管线。ESP32 按住按键说话时会发送多个 Opus 音频帧，当前每帧分别识别和回复，导致乱序错误。

**目标：** 实现 press-to-talk 状态机——`key_down`/`wake_word` 开始缓冲，二进制帧追加到缓冲区，`key_up` 合并音频后触发一次管线。

## 设计决策

| 问题 | 决策 |
|------|------|
| SessionState 存储位置 | `websocket_chat` 局部变量，传引用给 handler（最简单，无需清理） |
| wake_word vs key_down | 行为一致，均触发 idle→listening 转换 |
| 最大缓冲 | 10MB（`MAX_AUDIO_BUFFER_BYTES`），超限 drop 帧并 log warning |
| 空音频 key_up | Log warning，跳过管线，直接回 idle |
| idle 状态收到二进制帧 | Drop + log warning（严格状态机） |
| processing 状态收到二进制帧 | Drop + log warning |
| 管线异常 | `finally` 块保证 state 回 idle |

## 状态机

```
idle ──(key_down/wake_word)──► listening ──(key_up)──► processing ──(done/error)──► idle
         send "listening"       append audio            drain → STT→LLM→TTS
```

## 修改文件

### 1. `tests/test_main.py` — 先写失败测试（TDD）

**新增测试（8 个）：**

| 测试 | 类型 | 验证点 |
|------|------|--------|
| `test_session_state_defaults` | 单元 | status=="idle", audio_chunks==[] |
| `test_key_down_starts_listening` | 集成 | key_down → 返回 listening 状态 |
| `test_key_up_triggers_pipeline` | 集成 | key_down → binary → key_up → thinking + text_reply + audio |
| `test_key_up_empty_buffer_skips_pipeline` | 集成 | key_down → key_up（无音频）→ 不触发管线，连接存活 |
| `test_binary_frame_while_idle_is_dropped` | 集成 | idle 状态发 binary → 不触发管线，连接存活 |
| `test_binary_frames_are_concatenated` | 集成 | 3 个 binary 帧合并后传给 process_stt |
| `test_repeated_key_down_resets_buffer` | 集成 | key_down → binary → key_down → binary → key_up → 只有第二次的音频 |
| `test_key_up_without_key_down_is_ignored` | 集成 | idle 状态发 key_up → 不崩溃，连接存活 |

**修改 1 个现有测试：**
- `test_websocket_binary_frame_full_pipeline` → 改为 key_down → binary → key_up 序列

### 2. `main.py` — 实现变更

#### 2a. 新增导入

```python
from typing import Literal  # 第 1-14 行区域
```

#### 2b. 新增常量（Configuration 区域）

```python
MAX_AUDIO_BUFFER_BYTES: int = 10 * 1024 * 1024  # 10 MB
```

#### 2c. 新增 `SessionState` 模型（Pydantic models 区域后，约第 65 行后）

```python
class SessionState(BaseModel):
    status: Literal["idle", "listening", "processing"] = "idle"
    audio_chunks: list[bytes] = []

    def buffer_size(self) -> int:
        return sum(len(c) for c in self.audio_chunks)

    def drain_audio(self) -> bytes:
        audio = b"".join(self.audio_chunks)
        self.audio_chunks.clear()
        return audio

    def reset(self) -> None:
        self.status = "idle"
        self.audio_chunks.clear()
```

#### 2d. 重构 `handle_text_frame`

签名变更：`(ws, raw)` → `(ws, raw, state, session_id)`

新增逻辑：
- `KEY_DOWN` / `WAKE_WORD`：idle→listening（send "listening"）；listening 时 reset buffer 重新开始
- `KEY_UP`：listening→processing，调用 `handle_pipeline`；idle 时 ignore

#### 2e. 重构 `handle_binary_frame`

签名变更：`(ws, data, session_id)` → `(ws, data, state)`

新逻辑：仅在 listening 状态 append，其余状态 drop。检查 buffer 大小上限。
**不再调用 process_stt/call_openclaw/process_tts。**

#### 2f. 新增 `handle_pipeline(ws, state, session_id)`

从旧 `handle_binary_frame` 提取 STT→LLM→TTS 逻辑：
1. `audio = state.drain_audio()`
2. 空音频 → log + reset + return
3. `try:` 执行 STT → thinking → heartbeat + LLM → text_reply → TTS → binary
4. `finally: state.reset()` 保证回 idle

#### 2g. 更新 `websocket_chat`

```python
state = SessionState()  # 新增
# 传 state 给 handler：
await handle_text_frame(ws, text_data, state, session_id)
await handle_binary_frame(ws, byte_data, state)
```

### 3. `test_client.py` — 更新手动测试脚本

改为 key_down → binary → key_up 序列。

## 不变的部分

- `process_stt`, `call_openclaw`, `process_tts` — 内部不变
- Pydantic 消息模型（HardwareAction/Event, StateNotification, TextReply）— 不变
- `/health` 端点 — 不变
- 6 个管线单元测试 — 不变
- requirements.txt — 不变

## 验证

```bash
source venv/bin/activate && python -m pytest tests/ -v
# 预期：~19 个测试全部通过
```

+# Plan: Phase 1 — Audio Buffering (Press-to-Talk)

-openclaw-proxy Phase 0.5 已跑通：ESP32 ↔ WebSocket ↔ OpenClaw CLI 对话链路完整，但 STT 和 TTS 仍是 mock。

-现在需要接入真实的 Whisper（STT）和 edge-tts（TTS），让语音输入→文字→语音输出全链路打通。

+Phase 0/0.5/2/4 已完成：WebSocket 握手、OpenClaw CLI 对接、真实 STT (Whisper) 和 TTS (edge-tts) 均已跑通，11 个测试全部通过。

-**环境确认（均已安装）：**

-- openai-whisper 20250625 + PyTorch 2.10.0 (CPU)

-- ffmpeg 7.0.2 (libopus encoder/decoder)

+**当前问题：** 每个二进制帧独立触发完整 STT→LLM→TTS 管线。ESP32 按住按键说话时会发送多个 Opus 音频帧，当前每帧分别识别和回复，导致乱序错误。

+**目标：** 实现 press-to-talk 状态机——`key_down`/`wake_word` 开始缓冲，二进制帧追加到缓冲区，`key_up` 合并音频后触发一次管线。

-### 1. STT — `process_stt(audio_bytes) -> str`

-**链路：** Opus bytes → ffmpeg → 16kHz WAV → Whisper base 模型 → 文本

+| 问题 | 决策 |

+| SessionState 存储位置 | `websocket_chat` 局部变量，传引用给 handler（最简单，无需清理） |

+| wake_word vs key_down | 行为一致，均触发 idle→listening 转换 |

+| 最大缓冲 | 10MB（`MAX_AUDIO_BUFFER_BYTES`），超限 drop 帧并 log warning |

+| 空音频 key_up | Log warning，跳过管线，直接回 idle |

+| idle 状态收到二进制帧 | Drop + log warning（严格状态机） |

+| processing 状态收到二进制帧 | Drop + log warning |

-- 模块级加载 Whisper base 模型（一次加载，全局复用）

-- 用 `asyncio.to_thread()` 包装 Whisper 推理（CPU 密集，不能阻塞事件循环）

-- Whisper 的 `load_audio()` 内部会调 ffmpeg 处理任意格式，所以只需把 Opus 写入临时文件传路径即可

-- 用 `tempfile.NamedTemporaryFile` 写 Opus 字节，传路径给 `model.transcribe()`

-**关键代码骨架：**

-async def process_stt(audio_bytes: bytes) -> str:

-    with tempfile.NamedTemporaryFile(suffix=".opus", delete=True) as f:

-        result = await asyncio.to_thread(

-            _whisper_model.transcribe, f.name, language="zh"

+idle ──(key_down/wake_word)──► listening ──(key_up)──► processing ──(done/error)──► idle

+         send "listening"       append audio            drain → STT→LLM→TTS

-**修改文件：** `main.py` 第 54-86 行（`process_stt` 函数）+ 模块顶部导入

-### 2. TTS — `process_tts(text) -> bytes`

+### 1. `tests/test_main.py` — 先写失败测试（TDD）

-**链路：** 文本 → edge-tts (MP3) → ffmpeg → Opus bytes

+**新增测试（8 个）：**

-- edge-tts 是异步的，用 `Communicate(text, voice).stream()` 获取 MP3 流

-- 收集完整 MP3 字节后，用 ffmpeg subprocess 转 Opus：

-  `ffmpeg -i pipe:0 -c:a libopus -b:a 24k -ar 16000 -ac 1 -f opus pipe:1`

-- 返回完整 Opus 字节

+| 测试 | 类型 | 验证点 |

+| `test_session_state_defaults` | 单元 | status=="idle", audio_chunks==[] |

+| `test_key_down_starts_listening` | 集成 | key_down → 返回 listening 状态 |

+| `test_key_up_triggers_pipeline` | 集成 | key_down → binary → key_up → thinking + text_reply + audio |

+| `test_key_up_empty_buffer_skips_pipeline` | 集成 | key_down → key_up（无音频）→ 不触发管线，连接存活 |

+| `test_binary_frame_while_idle_is_dropped` | 集成 | idle 状态发 binary → 不触发管线，连接存活 |

+| `test_binary_frames_are_concatenated` | 集成 | 3 个 binary 帧合并后传给 process_stt |

+| `test_repeated_key_down_resets_buffer` | 集成 | key_down → binary → key_down → binary → key_up → 只有第二次的音频 |

+| `test_key_up_without_key_down_is_ignored` | 集成 | idle 状态发 key_up → 不崩溃，连接存活 |

-**关键代码骨架：**

+**修改 1 个现有测试：**

+- `test_websocket_binary_frame_full_pipeline` → 改为 key_down → binary → key_up 序列

-async def process_tts(text: str) -> bytes:

-    communicate = edge_tts.Communicate(text=text, voice=EDGE_TTS_VOICE)

-    async for chunk in communicate.stream():

-            mp3_chunks.append(chunk["data"])

-    proc = await asyncio.create_subprocess_exec(

-        stdout=asyncio.subprocess.PIPE,

-        stderr=asyncio.subprocess.PIPE,

-    opus_bytes, _ = await proc.communicate(input=mp3_bytes)

+from typing import Literal  # 第 1-14 行区域

-**修改文件：** `main.py` 第 100-122 行（`process_tts` 函数）+ 模块顶部导入

+MAX_AUDIO_BUFFER_BYTES: int = 10 * 1024 * 1024  # 10 MB

+#### 2c. 新增 `SessionState` 模型（Pydantic models 区域后，约第 65 行后）

-FFMPEG_BIN = which("ffmpeg") or "ffmpeg"

+    status: Literal["idle", "listening", "processing"] = "idle"

+        return sum(len(c) for c in self.audio_chunks)

+        audio = b"".join(self.audio_chunks)

+签名变更：`(ws, raw)` → `(ws, raw, state, session_id)`

+- `KEY_DOWN` / `WAKE_WORD`：idle→listening（send "listening"）；listening 时 reset buffer 重新开始

+- `KEY_UP`：listening→processing，调用 `handle_pipeline`；idle 时 ignore

+签名变更：`(ws, data, session_id)` → `(ws, data, state)`

+新逻辑：仅在 listening 状态 append，其余状态 drop。检查 buffer 大小上限。

+#### 2f. 新增 `handle_pipeline(ws, state, session_id)`

+从旧 `handle_binary_frame` 提取 STT→LLM→TTS 逻辑：

+3. `try:` 执行 STT → thinking → heartbeat + LLM → text_reply → TTS → binary

+await handle_text_frame(ws, text_data, state, session_id)

+await handle_binary_frame(ws, byte_data, state)

-- `test_process_stt` — mock `whisper.load_model` 和 `model.transcribe`，验证 tempfile 和结果解析

-- `test_process_tts` — mock `edge_tts.Communicate` 和 ffmpeg subprocess，验证 MP3→Opus 管线

-- 保持现有 10 个测试不变（mock subprocess 的 call_openclaw 测试不受影响）

+- `process_stt`, `call_openclaw`, `process_tts` — 内部不变

+- Pydantic 消息模型（HardwareAction/Event, StateNotification, TextReply）— 不变

+- 6 个管线单元测试 — 不变

-# 3. 用 test_client.py 测试完整链路（mock Opus → real STT → real LLM → real TTS → Opus）

+# 预期：~19 个测试全部通过

# Plan: Real STT + TTS Integration

## Context

openclaw-proxy Phase 0.5 已跑通：ESP32 ↔ WebSocket ↔ OpenClaw CLI 对话链路完整，但 STT 和 TTS 仍是 mock。
现在需要接入真实的 Whisper（STT）和 edge-tts（TTS），让语音输入→文字→语音输出全链路打通。

**环境确认（均已安装）：**
- openai-whisper 20250625 + PyTorch 2.10.0 (CPU)
- edge-tts 7.2.7
- ffmpeg 7.0.2 (libopus encoder/decoder)
- 4GB 可用内存，4 核 CPU，无 GPU

## 实现方案

### 1. STT — `process_stt(audio_bytes) -> str`

**链路：** Opus bytes → ffmpeg → 16kHz WAV → Whisper base 模型 → 文本

**实现：**
- 模块级加载 Whisper base 模型（一次加载，全局复用）
- 用 `asyncio.to_thread()` 包装 Whisper 推理（CPU 密集，不能阻塞事件循环）
- Whisper 的 `load_audio()` 内部会调 ffmpeg 处理任意格式，所以只需把 Opus 写入临时文件传路径即可
- 用 `tempfile.NamedTemporaryFile` 写 Opus 字节，传路径给 `model.transcribe()`

**关键代码骨架：**
```python
import whisper
_whisper_model = whisper.load_model("base")

async def process_stt(audio_bytes: bytes) -> str:
    with tempfile.NamedTemporaryFile(suffix=".opus", delete=True) as f:
        f.write(audio_bytes)
        f.flush()
        result = await asyncio.to_thread(
            _whisper_model.transcribe, f.name, language="zh"
        )
    return result["text"].strip()
```

**修改文件：** `main.py` 第 54-86 行（`process_stt` 函数）+ 模块顶部导入

### 2. TTS — `process_tts(text) -> bytes`

**链路：** 文本 → edge-tts (MP3) → ffmpeg → Opus bytes

**实现：**
- edge-tts 是异步的，用 `Communicate(text, voice).stream()` 获取 MP3 流
- 收集完整 MP3 字节后，用 ffmpeg subprocess 转 Opus：
  `ffmpeg -i pipe:0 -c:a libopus -b:a 24k -ar 16000 -ac 1 -f opus pipe:1`
- 返回完整 Opus 字节

**关键代码骨架：**
```python
import edge_tts
EDGE_TTS_VOICE = "zh-CN-XiaoxiaoNeural"

async def process_tts(text: str) -> bytes:
    # 1. edge-tts → MP3
    communicate = edge_tts.Communicate(text=text, voice=EDGE_TTS_VOICE)
    mp3_chunks = []
    async for chunk in communicate.stream():
        if chunk["type"] == "audio":
            mp3_chunks.append(chunk["data"])
    mp3_bytes = b"".join(mp3_chunks)

    # 2. MP3 → Opus via ffmpeg
    proc = await asyncio.create_subprocess_exec(
        "ffmpeg", "-i", "pipe:0", "-c:a", "libopus",
        "-b:a", "24k", "-ar", "16000", "-ac", "1", "-f", "opus", "pipe:1",
        stdin=asyncio.subprocess.PIPE,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    opus_bytes, _ = await proc.communicate(input=mp3_bytes)
    return opus_bytes
```

**修改文件：** `main.py` 第 100-122 行（`process_tts` 函数）+ 模块顶部导入

### 3. 配置常量

在 `main.py` 顶部 Configuration 区域新增：
```python
WHISPER_MODEL_SIZE = "base"
EDGE_TTS_VOICE = "zh-CN-XiaoxiaoNeural"
FFMPEG_BIN = which("ffmpeg") or "ffmpeg"
```

### 4. requirements.txt 更新

新增：
```
openai-whisper>=20250625
edge-tts>=7.0
```
（系统已安装，写入 requirements.txt 声明依赖）

### 5. 测试更新

- `test_process_stt` — mock `whisper.load_model` 和 `model.transcribe`，验证 tempfile 和结果解析
- `test_process_tts` — mock `edge_tts.Communicate` 和 ffmpeg subprocess，验证 MP3→Opus 管线
- 保持现有 10 个测试不变（mock subprocess 的 call_openclaw 测试不受影响）

## 不改的东西

- `.openclaw/` 目录 — 不动
- `call_openclaw()` — 已经是真实 CLI 调用，不动
- WebSocket 协议 / 帧路由 / 心跳逻辑 — 不动

## 验证方式

```bash
# 1. 跑单元测试
source venv/bin/activate && python -m pytest tests/ -v

# 2. 启动服务
python main.py

# 3. 用 test_client.py 测试完整链路（mock Opus → real STT → real LLM → real TTS → Opus）
python test_client.py
```


ExitPlanMode

Prompt: run tests

# Plan: Phase 1 — Audio Buffering (Press-to-Talk)

## Context

Phase 0/0.5/2/4 已完成：WebSocket 握手、OpenClaw CLI 对接、真实 STT (Whisper) 和 TTS (edge-tts) 均已跑通，11 个测试全部通过。

**当前问题：** 每个二进制帧独立触发完整 STT→LLM→TTS 管线。ESP32 按住按键说话时会发送多个 Opus 音频帧，当前每帧分别识别和回复，导致乱序错误。

**目标：** 实现 press-to-talk 状态机——`key_down`/`wake_word` 开始缓冲，二进制帧追加到缓冲区，`key_up` 合并音频后触发一次管线。

## 设计决策

| 问题 | 决策 |
|------|------|
| SessionState 存储位置 | `websocket_chat` 局部变量，传引用给 handler（最简单，无需清理） |
| wake_word vs key_down | 行为一致，均触发 idle→listening 转换 |
| 最大缓冲 | 10MB（`MAX_AUDIO_BUFFER_BYTES`），超限 drop 帧并 log warning |
| 空音频 key_up | Log warning，跳过管线，直接回 idle |
| idle 状态收到二进制帧 | Drop + log warning（严格状态机） |
| processing 状态收到二进制帧 | Drop + log warning |
| 管线异常 | `finally` 块保证 state 回 idle |

## 状态机

```
idle ──(key_down/wake_word)──► listening ──(key_up)──► processing ──(done/error)──► idle
         send "listening"       append audio            drain → STT→LLM→TTS
```

## 修改文件

### 1. `tests/test_main.py` — 先写失败测试（TDD）

**新增测试（8 个）：**

| 测试 | 类型 | 验证点 |
|------|------|--------|
| `test_session_state_defaults` | 单元 | status=="idle", audio_chunks==[] |
| `test_key_down_starts_listening` | 集成 | key_down → 返回 listening 状态 |
| `test_key_up_triggers_pipeline` | 集成 | key_down → binary → key_up → thinking + text_reply + audio |
| `test_key_up_empty_buffer_skips_pipeline` | 集成 | key_down → key_up（无音频）→ 不触发管线，连接存活 |
| `test_binary_frame_while_idle_is_dropped` | 集成 | idle 状态发 binary → 不触发管线，连接存活 |
| `test_binary_frames_are_concatenated` | 集成 | 3 个 binary 帧合并后传给 process_stt |
| `test_repeated_key_down_resets_buffer` | 集成 | key_down → binary → key_down → binary → key_up → 只有第二次的音频 |
| `test_key_up_without_key_down_is_ignored` | 集成 | idle 状态发 key_up → 不崩溃，连接存活 |

**修改 1 个现有测试：**
- `test_websocket_binary_frame_full_pipeline` → 改为 key_down → binary → key_up 序列

### 2. `main.py` — 实现变更

#### 2a. 新增导入

```python
from typing import Literal  # 第 1-14 行区域
```

#### 2b. 新增常量（Configuration 区域）

```python
MAX_AUDIO_BUFFER_BYTES: int = 10 * 1024 * 1024  # 10 MB
```

#### 2c. 新增 `SessionState` 模型（Pydantic models 区域后，约第 65 行后）

```python
class SessionState(BaseModel):
    status: Literal["idle", "listening", "processing"] = "idle"
    audio_chunks: list[bytes] = []

    def buffer_size(self) -> int:
        return sum(len(c) for c in self.audio_chunks)

    def drain_audio(self) -> bytes:
        audio = b"".join(self.audio_chunks)
        self.audio_chunks.clear()
        return audio

    def reset(self) -> None:
        self.status = "idle"
        self.audio_chunks.clear()
```

#### 2d. 重构 `handle_text_frame`

签名变更：`(ws, raw)` → `(ws, raw, state, session_id)`

新增逻辑：
- `KEY_DOWN` / `WAKE_WORD`：idle→listening（send "listening"）；listening 时 reset buffer 重新开始
- `KEY_UP`：listening→processing，调用 `handle_pipeline`；idle 时 ignore

#### 2e. 重构 `handle_binary_frame`

签名变更：`(ws, data, session_id)` → `(ws, data, state)`

新逻辑：仅在 listening 状态 append，其余状态 drop。检查 buffer 大小上限。
**不再调用 process_stt/call_openclaw/process_tts。**

#### 2f. 新增 `handle_pipeline(ws, state, session_id)`

从旧 `handle_binary_frame` 提取 STT→LLM→TTS 逻辑：
1. `audio = state.drain_audio()`
2. 空音频 → log + reset + return
3. `try:` 执行 STT → thinking → heartbeat + LLM → text_reply → TTS → binary
4. `finally: state.reset()` 保证回 idle

#### 2g. 更新 `websocket_chat`

```python
state = SessionState()  # 新增
# 传 state 给 handler：
await handle_text_frame(ws, text_data, state, session_id)
await handle_binary_frame(ws, byte_data, state)
```

### 3. `test_client.py` — 更新手动测试脚本

改为 key_down → binary → key_up 序列。

## 不变的部分

- `process_stt`, `call_openclaw`, `process_tts` — 内部不变
- Pydantic 消息模型（HardwareAction/Event, StateNotification, TextReply）— 不变
- `/health` 端点 — 不变
- 6 个管线单元测试 — 不变
- requirements.txt — 不变

## 验证

```bash
source venv/bin/activate && python -m pytest tests/ -v
# 预期：~19 个测试全部通过
```