Design the implementation plan for Phase 1 (Audio Buffering / press-to-talk) of the openclaw-proxy project.

## Current Architecture

**main.py** (309 lines) has these key components:

### Pydantic Models (lines 34-64):
- `HardwareAction(str, Enum)`: KEY_DOWN, KEY_UP, WAKE_WORD
- `HardwareEvent(BaseModel)`: type + action
- `StateNotification(BaseModel)`: type="state" + status
- `TextReply(BaseModel)`: type="text_reply" + text

### Frame Routing:
- `handle_text_frame(ws, raw)` (lines 193-216): Parses JSON, validates HardwareEvent. Only handles WAKE_WORD (sends "listening" state). KEY_DOWN and KEY_UP are parsed but **ignored**.
- `handle_binary_frame(ws, data, session_id)` (lines 219-261): Each binary frame **independently** triggers the full STT→LLM→TTS pipeline. This is the core problem.

### WebSocket endpoint (lines 269-294):
```python
async def websocket_chat(ws: WebSocket) -> None:
    await ws.accept()
    session_id = f"xiaozhi-{uuid.uuid4().hex[:8]}"
    try:
        while True:
            message = await ws.receive()
            if message["type"] == "websocket.disconnect": break
            text_data = message.get("text")
            byte_data = message.get("bytes")
            if text_data is not None:
                await handle_text_frame(ws, text_data)
            elif byte_data is not None:
                await handle_binary_frame(ws, byte_data, session_id)
    except WebSocketDisconnect: pass
```

No per-connection state exists beyond `session_id`.

## Phase 1 Requirements (from PLAN.md)

1. Implement `SessionState` per connection:
```python
class SessionState(BaseModel):
    status: Literal["idle", "listening", "processing"] = "idle"
    audio_chunks: list[bytes] = []
```

2. State machine:
```
idle ─(key_down/wake_word)─> listening ─(key_up)─> processing ─(done)─> idle
```

3. Binary frames in "listening" state: append to audio_chunks buffer
4. key_up: join all audio_chunks, pass complete audio to pipeline
5. Refactor handle_binary_frame to just append
6. New handle_pipeline(ws, session_state, session_id) for unified STT→LLM→TTS
7. Edge cases: empty audio, oversized audio (need max buffer), repeated key_down, key_up without key_down, binary while idle

## Design Constraints

- CLAUDE.md rules: TDD, type hints mandatory, Pydantic models, Google-style docstrings
- Keep all changes in main.py (no module split yet per PLAN.md)
- Don't modify process_stt, call_openclaw, process_tts internals
- Don't modify the WebSocket protocol or health endpoint
- Existing 11 tests must continue passing (may need mock updates)

## Questions to Address in the Plan

1. Where to store SessionState? Options:
   a. Local variable in websocket_chat (simplest, passed to handlers)
   b. Module-level dict keyed by ws or session_id

2. How to handle wake_word vs key_down? Both should start listening. PLAN.md says both trigger buffering.

3. What max buffer size? Suggest 10MB or 60 seconds of audio.

4. What to do with empty audio on key_up? Log warning and skip pipeline.

5. What to do with binary frames while in "idle" state? Options:
   a. Drop silently (strict state machine)
   b. Auto-enter listening and buffer (lenient)

   I recommend option (a) - drop with a warning log. The ESP32 firmware should send key_down first.

6. How to handle binary frames during "processing" state? Drop them - can't start new recording while pipeline is running.

7. Test strategy: What new tests are needed?

Please provide a detailed step-by-step implementation plan with code changes needed for each file.


I'll start by thoroughly exploring the codebase to understand the current architecture, test structure, and existing patterns before designing the implementation plan.


Bash

List project root contents


ls -la /root/projects/openclaw-proxy/tests/ 2>/dev/null || echo "No tests directory"

List test directory contents


total 12
drwxr-xr-x 3 root root   64 Mar 16 00:13 .
drwxr-xr-x 7 root root 4096 Mar 16 00:28 ..
-rw-r--r-- 1 root root    0 Mar 15 21:40 __init__.py
drwxr-xr-x 2 root root   84 Mar 16 00:14 __pycache__
-rw-r--r-- 1 root root 7463 Mar 16 00:13 test_main.py


total 60
drwxr-xr-x 7 root root   4096 Mar 16 00:28 .
drwxr-xr-x 6  501 games    82 Mar 15 21:31 ..
drwxr-xr-x 2 root root     33 Mar 16 00:23 .claude
-rw-r--r-- 1 root root   1392 Mar 15 21:33 CLAUDE.md
-rw-r--r-- 1 root root   9972 Mar 16 00:14 main.py
-rw-r--r-- 1 root root  20939 Mar 16 00:27 MILESTONE-STT-TTS.md
-rw-r--r-- 1 root root   7663 Mar 16 00:28 PLAN.md
drwxr-xr-x 2 root root     34 Mar 16 00:14 __pycache__
drwxr-xr-x 3 root root     70 Mar 15 21:43 .pytest_cache
-rw-r--r-- 1 root root    162 Mar 16 00:14 requirements.txt
-rw-r--r-- 1 root root   1077 Mar 15 23:53 test_client.py
drwxr-xr-x 3 root root     64 Mar 16 00:13 tests
drwxr-xr-x 6 root root     87 Mar 16 00:19 venv


     1→"""Tests for openclaw-proxy WebSocket middleware."""
     2→
     3→import json
     4→from unittest.mock import AsyncMock, MagicMock, patch
     5→
     6→import pytest
     7→from starlette.testclient import TestClient
     8→
     9→from main import app, process_stt, call_openclaw, process_tts
    10→
    11→
    12→# ── Helpers ──────────────────────────────────────────────────────────
    13→
    14→MOCK_OPENCLAW_OUTPUT = json.dumps({
    15→    "status": "ok",
    16→    "result": {
    17→        "payloads": [{"text": "你好！我在这儿。", "mediaUrl": None}],
    18→    },
    19→}).encode()
    20→
    21→
    22→def _make_mock_process(
    23→    stdout: bytes = MOCK_OPENCLAW_OUTPUT,
    24→    returncode: int = 0,
    25→) -> AsyncMock:
    26→    """Build a mock asyncio.Process with preset stdout/stderr."""
    27→    proc = AsyncMock()
    28→    proc.communicate.return_value = (stdout, b"")
    29→    proc.returncode = returncode
    30→    return proc
    31→
    32→
    33→# ── Unit tests: pipeline functions ───────────────────────────────────
    34→
    35→
    36→@pytest.mark.asyncio
    37→async def test_process_stt_calls_whisper() -> None:
    38→    """STT should write audio to temp file, call Whisper, and return stripped text."""
    39→    mock_model = MagicMock()
    40→    mock_model.transcribe.return_value = {"text": " 你好世界 "}
    41→    with patch("main._get_whisper_model", return_value=mock_model):
    42→        result = await process_stt(b"\x00" * 80)
    43→    assert result == "你好世界"
    44→    mock_model.transcribe.assert_called_once()
    45→    args, kwargs = mock_model.transcribe.call_args
    46→    assert args[0].endswith(".opus")
    47→    assert kwargs == {"language": "zh"}
    48→
    49→
    50→@pytest.mark.asyncio
    51→async def test_call_openclaw_parses_cli_output() -> None:
    52→    """call_openclaw should parse the CLI JSON and return reply text."""
    53→    mock_proc = _make_mock_process()
    54→    with patch("main.asyncio.create_subprocess_exec", return_value=mock_proc):
    55→        result = await call_openclaw("你好", "test-session")
    56→    assert result == "你好！我在这儿。"
    57→
    58→
    59→@pytest.mark.asyncio
    60→async def test_call_openclaw_raises_on_cli_failure() -> None:
    61→    """call_openclaw should raise RuntimeError when CLI exits non-zero."""
    62→    mock_proc = _make_mock_process(stdout=b"", returncode=1)
    63→    mock_proc.communicate.return_value = (b"", b"some error")
    64→    with patch("main.asyncio.create_subprocess_exec", return_value=mock_proc):
    65→        with pytest.raises(RuntimeError, match="OpenClaw CLI error"):
    66→            await call_openclaw("你好", "test-session")
    67→
    68→
    69→@pytest.mark.asyncio
    70→async def test_call_openclaw_raises_on_empty_payloads() -> None:
    71→    """call_openclaw should raise RuntimeError on empty payloads."""
    72→    empty = json.dumps({"status": "ok", "result": {"payloads": []}}).encode()
    73→    mock_proc = _make_mock_process(stdout=empty)
    74→    with patch("main.asyncio.create_subprocess_exec", return_value=mock_proc):
    75→        with pytest.raises(RuntimeError, match="empty payloads"):
    76→            await call_openclaw("你好", "test-session")
    77→
    78→
    79→@pytest.mark.asyncio
    80→async def test_process_tts_returns_opus_bytes() -> None:
    81→    """TTS should collect MP3 from edge-tts, convert to Opus via ffmpeg."""
    82→
    83→    async def mock_stream() -> None:  # type: ignore[override]
    84→        yield {"type": "audio", "data": b"mp3-part1"}
    85→        yield {"type": "audio", "data": b"mp3-part2"}
    86→        yield {"type": "WordBoundary", "data": None}  # non-audio chunk
    87→
    88→    mock_comm = MagicMock()
    89→    mock_comm.stream = mock_stream
    90→
    91→    mock_proc = AsyncMock()
    92→    mock_proc.communicate.return_value = (b"fake-opus", b"")
    93→    mock_proc.returncode = 0
    94→
    95→    with (
    96→        patch("main.edge_tts.Communicate", return_value=mock_comm),
    97→        patch("main.asyncio.create_subprocess_exec", return_value=mock_proc),
    98→    ):
    99→        result = await process_tts("测试文本")
   100→
   101→    assert result == b"fake-opus"
   102→    mock_proc.communicate.assert_called_once_with(input=b"mp3-part1mp3-part2")
   103→
   104→
   105→@pytest.mark.asyncio
   106→async def test_process_tts_raises_on_ffmpeg_failure() -> None:
   107→    """TTS should raise RuntimeError when ffmpeg exits non-zero."""
   108→
   109→    async def mock_stream() -> None:  # type: ignore[override]
   110→        yield {"type": "audio", "data": b"mp3-data"}
   111→
   112→    mock_comm = MagicMock()
   113→    mock_comm.stream = mock_stream
   114→
   115→    mock_proc = AsyncMock()
   116→    mock_proc.communicate.return_value = (b"", b"ffmpeg error msg")
   117→    mock_proc.returncode = 1
   118→
   119→    with (
   120→        patch("main.edge_tts.Communicate", return_value=mock_comm),
   121→        patch("main.asyncio.create_subprocess_exec", return_value=mock_proc),
   122→    ):
   123→        with pytest.raises(RuntimeError, match="ffmpeg TTS conversion error"):
   124→            await process_tts("测试文本")
   125→
   126→
   127→# ── Integration tests: HTTP & WebSocket ──────────────────────────────
   128→
   129→
   130→def test_health_endpoint() -> None:
   131→    """GET /health should return 200 with ok status."""
   132→    client = TestClient(app)
   133→    resp = client.get("/health")
   134→    assert resp.status_code == 200
   135→    assert resp.json() == {"status": "ok"}
   136→
   137→
   138→def test_websocket_connect_and_disconnect() -> None:
   139→    """Client should be able to connect to /chat and cleanly disconnect."""
   140→    client = TestClient(app)
   141→    with client.websocket_connect("/chat"):
   142→        pass
   143→
   144→
   145→def test_websocket_text_frame_wake_word() -> None:
   146→    """Sending a wake_word JSON should get back a 'listening' state."""
   147→    client = TestClient(app)
   148→    with client.websocket_connect("/chat") as ws:
   149→        ws.send_text(json.dumps({
   150→            "type": "hardware_event",
   151→            "action": "wake_word",
   152→        }))
   153→        resp = json.loads(ws.receive_text())
   154→        assert resp["type"] == "state"
   155→        assert resp["status"] == "listening"
   156→
   157→
   158→def test_websocket_text_frame_malformed_json() -> None:
   159→    """Malformed JSON should not crash the connection."""
   160→    client = TestClient(app)
   161→    with client.websocket_connect("/chat") as ws:
   162→        ws.send_text("not-valid-json{{{")
   163→        ws.send_text(json.dumps({
   164→            "type": "hardware_event",
   165→            "action": "wake_word",
   166→        }))
   167→        resp = json.loads(ws.receive_text())
   168→        assert resp["status"] == "listening"
   169→
   170→
   171→def test_websocket_binary_frame_full_pipeline() -> None:
   172→    """Binary audio should trigger STT → LLM → TTS pipeline.
   173→
   174→    Expected response sequence:
   175→      1. {"type": "state", "status": "thinking"}
   176→      2. {"type": "text_reply", "text": "..."}
   177→      3. binary bytes (TTS audio)
   178→    """
   179→    client = TestClient(app)
   180→    with (
   181→        patch(
   182→            "main.process_stt",
   183→            new_callable=AsyncMock,
   184→            return_value="测试语音输入",
   185→        ),
   186→        patch(
   187→            "main.call_openclaw",
   188→            new_callable=AsyncMock,
   189→            return_value="你好！我在这儿。",
   190→        ),
   191→        patch(
   192→            "main.process_tts",
   193→            new_callable=AsyncMock,
   194→            return_value=b"\xfc\xff\xfe",
   195→        ),
   196→    ):
   197→        with client.websocket_connect("/chat") as ws:
   198→            ws.send_bytes(b"\x00" * 80)
   199→
   200→            # 1) thinking state
   201→            msg1 = json.loads(ws.receive_text())
   202→            assert msg1 == {"type": "state", "status": "thinking"}
   203→
   204→            # 2) text reply
   205→            msg2 = json.loads(ws.receive_text())
   206→            assert msg2["type"] == "text_reply"
   207→            assert msg2["text"] == "你好！我在这儿。"
   208→
   209→            # 3) TTS audio bytes
   210→            audio = ws.receive_bytes()
   211→            assert audio == b"\xfc\xff\xfe"
   212→


     1→"""openclaw-proxy — async WebSocket gateway between XiaoZhi ESP32 and OpenClaw agent."""
     2→
     3→import asyncio
     4→import json
     5→import logging
     6→import tempfile
     7→import uuid
     8→from enum import Enum
     9→from shutil import which
    10→
    11→import edge_tts
    12→import whisper
    13→from fastapi import FastAPI, WebSocket, WebSocketDisconnect
    14→from pydantic import BaseModel, ValidationError
    15→
    16→# ── Logging ──────────────────────────────────────────────────────────
    17→
    18→logging.basicConfig(
    19→    level=logging.INFO,
    20→    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
    21→)
    22→logger = logging.getLogger("openclaw-proxy")
    23→
    24→# ── Configuration ────────────────────────────────────────────────────
    25→
    26→OPENCLAW_CLI = which("openclaw") or "openclaw"
    27→OPENCLAW_AGENT_ID = "main"
    28→OPENCLAW_TIMEOUT = 120  # seconds
    29→WHISPER_MODEL_SIZE = "base"
    30→EDGE_TTS_VOICE = "zh-CN-XiaoxiaoNeural"
    31→FFMPEG_BIN = which("ffmpeg") or "ffmpeg"
    32→
    33→
    34→# ── Pydantic models ─────────────────────────────────────────────────
    35→
    36→
    37→class HardwareAction(str, Enum):
    38→    """Actions reported by XiaoZhi ESP32 hardware."""
    39→
    40→    KEY_DOWN = "key_down"
    41→    KEY_UP = "key_up"
    42→    WAKE_WORD = "wake_word"
    43→
    44→
    45→class HardwareEvent(BaseModel):
    46→    """Incoming JSON frame from hardware."""
    47→
    48→    type: str
    49→    action: HardwareAction
    50→
    51→
    52→class StateNotification(BaseModel):
    53→    """Outgoing state frame pushed to hardware (e.g. breathing LED trigger)."""
    54→
    55→    type: str = "state"
    56→    status: str
    57→
    58→
    59→class TextReply(BaseModel):
    60→    """Outgoing text reply carrying the LLM response."""
    61→
    62→    type: str = "text_reply"
    63→    text: str
    64→
    65→
    66→# ── Whisper model (lazy singleton) ─────────────────────────────────
    67→
    68→_whisper_model: whisper.Whisper | None = None
    69→
    70→
    71→def _get_whisper_model() -> whisper.Whisper:
    72→    """Return the Whisper model, loading it on first call."""
    73→    global _whisper_model
    74→    if _whisper_model is None:
    75→        logger.info("Loading Whisper '%s' model…", WHISPER_MODEL_SIZE)
    76→        _whisper_model = whisper.load_model(WHISPER_MODEL_SIZE)
    77→    return _whisper_model
    78→
    79→
    80→# ── STT / TTS pipeline ────────────────────────────────────────────
    81→
    82→
    83→async def process_stt(audio_bytes: bytes) -> str:
    84→    """Convert Opus audio to text via Whisper.
    85→
    86→    Args:
    87→        audio_bytes: Raw Opus-encoded audio from ESP32.
    88→
    89→    Returns:
    90→        Transcribed text string.
    91→    """
    92→    logger.info("STT: received %d bytes of Opus audio", len(audio_bytes))
    93→    model = _get_whisper_model()
    94→    with tempfile.NamedTemporaryFile(suffix=".opus", delete=True) as f:
    95→        f.write(audio_bytes)
    96→        f.flush()
    97→        result = await asyncio.to_thread(
    98→            model.transcribe, f.name, language="zh"
    99→        )
   100→    text = result["text"].strip()
   101→    logger.info("STT: transcribed → %s", text)
   102→    return text
   103→
   104→
   105→async def call_openclaw(text: str, session_id: str) -> str:
   106→    """Send text to OpenClaw agent via CLI and return its reply.
   107→
   108→    Args:
   109→        text: User utterance transcribed by STT.
   110→        session_id: Per-connection session ID for multi-turn conversation.
   111→
   112→    Returns:
   113→        Agent reply string.
   114→
   115→    Raises:
   116→        RuntimeError: If the CLI call fails or returns unexpected output.
   117→    """
   118→    logger.info("LLM: sending to OpenClaw → %s", text)
   119→
   120→    proc = await asyncio.create_subprocess_exec(
   121→        OPENCLAW_CLI, "agent",
   122→        "--agent", OPENCLAW_AGENT_ID,
   123→        "--session-id", session_id,
   124→        "-m", text,
   125→        "--json",
   126→        "--timeout", str(OPENCLAW_TIMEOUT),
   127→        stdout=asyncio.subprocess.PIPE,
   128→        stderr=asyncio.subprocess.PIPE,
   129→    )
   130→    stdout, stderr = await proc.communicate()
   131→
   132→    if proc.returncode != 0:
   133→        err_msg = stderr.decode(errors="replace").strip()
   134→        logger.error("OpenClaw CLI failed (rc=%d): %s", proc.returncode, err_msg)
   135→        raise RuntimeError(f"OpenClaw CLI error: {err_msg}")
   136→
   137→    data = json.loads(stdout.decode())
   138→    payloads = data.get("result", {}).get("payloads", [])
   139→    if not payloads:
   140→        raise RuntimeError("OpenClaw returned empty payloads")
   141→
   142→    reply = payloads[0].get("text", "")
   143→    logger.info("LLM: reply ← %s", reply)
   144→    return reply
   145→
   146→
   147→async def process_tts(text: str) -> bytes:
   148→    """Convert text to Opus audio via edge-tts and ffmpeg.
   149→
   150→    Args:
   151→        text: Text to synthesize.
   152→
   153→    Returns:
   154→        Opus-encoded audio bytes.
   155→
   156→    Raises:
   157→        RuntimeError: If ffmpeg conversion fails.
   158→    """
   159→    logger.info("TTS: synthesising → %s", text)
   160→
   161→    # 1. edge-tts → MP3
   162→    communicate = edge_tts.Communicate(text=text, voice=EDGE_TTS_VOICE)
   163→    mp3_chunks: list[bytes] = []
   164→    async for chunk in communicate.stream():
   165→        if chunk["type"] == "audio":
   166→            mp3_chunks.append(chunk["data"])
   167→    mp3_bytes = b"".join(mp3_chunks)
   168→
   169→    # 2. MP3 → Opus via ffmpeg
   170→    proc = await asyncio.create_subprocess_exec(
   171→        FFMPEG_BIN, "-i", "pipe:0",
   172→        "-c:a", "libopus", "-b:a", "24k",
   173→        "-ar", "16000", "-ac", "1",
   174→        "-f", "opus", "pipe:1",
   175→        stdin=asyncio.subprocess.PIPE,
   176→        stdout=asyncio.subprocess.PIPE,
   177→        stderr=asyncio.subprocess.PIPE,
   178→    )
   179→    opus_bytes, stderr = await proc.communicate(input=mp3_bytes)
   180→
   181→    if proc.returncode != 0:
   182→        err_msg = stderr.decode(errors="replace").strip()
   183→        logger.error("ffmpeg failed (rc=%d): %s", proc.returncode, err_msg)
   184→        raise RuntimeError(f"ffmpeg TTS conversion error: {err_msg}")
   185→
   186→    logger.info("TTS: generated %d bytes of Opus audio", len(opus_bytes))
   187→    return opus_bytes
   188→
   189→
   190→# ── Frame router ────────────────────────────────────────────────────
   191→
   192→
   193→async def handle_text_frame(ws: WebSocket, raw: str) -> None:
   194→    """Parse and route an incoming JSON text frame.
   195→
   196→    Args:
   197→        ws: Active WebSocket connection.
   198→        raw: Raw text payload received from client.
   199→    """
   200→    try:
   201→        payload = json.loads(raw)
   202→    except json.JSONDecodeError:
   203→        logger.warning("Malformed JSON (ignored): %.120s", raw)
   204→        return
   205→
   206→    try:
   207→        event = HardwareEvent(**payload)
   208→    except ValidationError as exc:
   209→        logger.warning("Invalid hardware event: %s", exc.errors())
   210→        return
   211→
   212→    logger.info("Hardware event: %s", event.action.value)
   213→
   214→    if event.action == HardwareAction.WAKE_WORD:
   215→        ack = StateNotification(status="listening")
   216→        await ws.send_text(ack.model_dump_json())
   217→
   218→
   219→async def handle_binary_frame(ws: WebSocket, data: bytes, session_id: str) -> None:
   220→    """Run the full STT → LLM → TTS pipeline on an audio binary frame.
   221→
   222→    Args:
   223→        ws: Active WebSocket connection.
   224→        data: Raw Opus audio bytes.
   225→        session_id: Per-connection session ID.
   226→
   227→    Response sequence pushed to client:
   228→        1. {"type": "state", "status": "thinking"}  — triggers breathing LED
   229→        2. {"type": "text_reply", "text": "..."}     — agent answer
   230→        3. <binary>                                   — Opus audio of the answer
   231→    """
   232→    # Stage 1 — STT
   233→    text = await process_stt(data)
   234→
   235→    # Notify hardware: entering thinking phase
   236→    await ws.send_text(StateNotification(status="thinking").model_dump_json())
   237→
   238→    # Stage 2 — LLM (with periodic thinking heartbeats)
   239→    async def heartbeat() -> None:
   240→        """Send thinking heartbeat every 3 seconds while LLM is working."""
   241→        while True:
   242→            await asyncio.sleep(3)
   243→            try:
   244→                await ws.send_text(
   245→                    StateNotification(status="thinking").model_dump_json()
   246→                )
   247→            except Exception:
   248→                break
   249→
   250→    heartbeat_task = asyncio.create_task(heartbeat())
   251→    try:
   252→        reply = await call_openclaw(text, session_id)
   253→    finally:
   254→        heartbeat_task.cancel()
   255→
   256→    await ws.send_text(TextReply(text=reply).model_dump_json())
   257→
   258→    # Stage 3 — TTS
   259→    audio = await process_tts(reply)
   260→    logger.info("Pushing %d bytes of audio to client", len(audio))
   261→    await ws.send_bytes(audio)
   262→
   263→
   264→# ── FastAPI application ─────────────────────────────────────────────
   265→
   266→app = FastAPI(title="openclaw-proxy", version="0.1.0")
   267→
   268→
   269→@app.websocket("/chat")
   270→async def websocket_chat(ws: WebSocket) -> None:
   271→    """Main WebSocket endpoint for XiaoZhi ESP32 hardware."""
   272→    await ws.accept()
   273→    peer = f"{ws.client.host}:{ws.client.port}" if ws.client else "unknown"
   274→    session_id = f"xiaozhi-{uuid.uuid4().hex[:8]}"
   275→    logger.info("Client connected: %s (session=%s)", peer, session_id)
   276→
   277→    try:
   278→        while True:
   279→            message = await ws.receive()
   280→
   281→            if message["type"] == "websocket.disconnect":
   282→                break
   283→
   284→            text_data = message.get("text")
   285→            byte_data = message.get("bytes")
   286→
   287→            if text_data is not None:
   288→                await handle_text_frame(ws, text_data)
   289→            elif byte_data is not None:
   290→                await handle_binary_frame(ws, byte_data, session_id)
   291→    except WebSocketDisconnect:
   292→        pass
   293→    finally:
   294→        logger.info("Client disconnected: %s (session=%s)", peer, session_id)
   295→
   296→
   297→@app.get("/health")
   298→async def health() -> dict[str, str]:
   299→    """Liveness probe."""
   300→    return {"status": "ok"}
   301→
   302→
   303→# ── Entry point ─────────────────────────────────────────────────────
   304→
   305→if __name__ == "__main__":
   306→    import uvicorn
   307→
   308→    uvicorn.run("main:app", host="0.0.0.0", port=8000, reload=True)
   309→


     1→# PLAN.md — openclaw-proxy
     2→
     3→## 项目定位
     4→
     5→轻量级异步 WebSocket 网关，部署在 Linux 云服务器上，负责在 **小智 ESP32 语音硬件** 与 **OpenClaw 智能体** (`/root/.openclaw/`) 之间做协议翻译和管线编排。
     6→
     7→```
     8→┌──────────────┐  Opus/JSON   ┌──────────────────┐   CLI    ┌──────────────┐
     9→│  XiaoZhi     │◄────────────►│  openclaw-proxy  │────────►│  OpenClaw    │
    10→│  ESP32       │  WebSocket   │  (FastAPI:8000)   │  stdout  │  Gateway     │
    11→└──────────────┘              └──────────────────┘◄────────│  (:18789)    │
    12→                                  │        ▲                └──────────────┘
    13→                                  ▼        │
    14→                              ┌──────┐ ┌──────┐
    15→                              │ STT  │ │ TTS  │
    16→                              └──────┘ └──────┘
    17→```
    18→
    19→**对接方式决策（2026-03-15）：** 采用方案 B — 通过 `openclaw agent` CLI 调用 gateway，
    20→不修改 `.openclaw/` 目录，不注册 channel 插件。优先跑通演示。
    21→
    22→## 协议约定
    23→
    24→### 上行（ESP32 → Proxy）
    25→
    26→| 帧类型 | 格式 | 说明 |
    27→|--------|------|------|
    28→| 文本帧 | `{"type": "hardware_event", "action": "wake_word\|key_down\|key_up"}` | 硬件状态事件 |
    29→| 二进制帧 | Raw Opus bytes | 用户语音音频流 |
    30→
    31→### 下行（Proxy → ESP32）
    32→
    33→| 帧类型 | 格式 | 说明 |
    34→|--------|------|------|
    35→| 文本帧 | `{"type": "state", "status": "listening\|thinking\|speaking"}` | 控制硬件 LED 状态 |
    36→| 文本帧 | `{"type": "text_reply", "text": "..."}` | LLM 文本回复 |
    37→| 二进制帧 | Raw Opus bytes | TTS 合成音频流 |
    38→
    39→## 分阶段实施计划
    40→
    41→---
    42→
    43→### Phase 0 — Mock 骨架搭建 ✅ 已完成
    44→
    45→**目标：** 跑通 WebSocket 握手和完整数据通路，用假数据验证时序正确。
    46→
    47→**交付物：**
    48→- [x] `main.py` — FastAPI + WS `/chat` 端点，帧路由，三段 mock 管线
    49→- [x] `tests/test_main.py` — 8 个测试全部通过（3 单元 + 5 集成）
    50→- [x] `requirements.txt` — 依赖声明
    51→
    52→---
    53→
    54→### Phase 0.5 — OpenClaw 真实对接 ✅ 已完成
    55→
    56→**目标：** 将 `call_openclaw` 从 mock 替换为真实 CLI 调用，跑通文本对话。
    57→
    58→**决策：** 采用 `openclaw agent --agent main --session-id {id} -m {text} --json` 命令。
    59→Gateway 运行在 `localhost:18789`，CLI 自动连接。
    60→
    61→**交付物：**
    62→- [x] `call_openclaw` 通过 `asyncio.create_subprocess_exec` 调用 CLI
    63→- [x] 每个 WebSocket 连接分配独立 `session_id`，支持多轮对话
    64→- [x] LLM 推理期间每 3 秒发送 thinking 心跳，保持硬件活跃
    65→- [x] CLI 失败 / 空 payload 的错误处理
    66→- [x] `tests/test_main.py` — 10 个测试全部通过（mock subprocess）
    67→
    68→**管线流程：**
    69→```
    70→Binary frame → process_stt(mock) → send "thinking" state
    71→            → call_openclaw(CLI) → heartbeat every 3s → send text_reply
    72→            → process_tts(mock) → send binary audio
    73→```
    74→
    75→---
    76→
    77→### Phase 1 — 音频缓冲与真实 ESP32 握手
    78→
    79→**目标：** 支持 ESP32 的真实交互模式——用户按住按键持续说话，松开后触发管线。
    80→
    81→**任务：**
    82→- [ ] 实现 `AudioBuffer` 类，按连接维护音频缓冲区
    83→  - `key_down` / `wake_word` → 开始缓冲，进入 "listening" 状态
    84→  - 后续二进制帧 → 追加到缓冲区
    85→  - `key_up` → 停止缓冲，将完整音频交给管线
    86→- [ ] 重构 `handle_binary_frame` 为缓冲追加逻辑
    87→- [ ] 新增 `handle_pipeline(ws, audio_buffer)` 统一编排三段管线
    88→- [ ] 处理边界情况：空音频、超长音频、重复 key_down
    89→- [ ] 补充对应测试用例
    90→
    91→**关键数据结构：**
    92→```python
    93→class SessionState(BaseModel):
    94→    status: Literal["idle", "listening", "processing"] = "idle"
    95→    audio_chunks: list[bytes] = []
    96→```
    97→
    98→---
    99→
   100→### Phase 2 — 真实 STT 接入 ✅ 已完成（2026-03-16）
   101→
   102→**目标：** 将 `process_stt` 从 mock 替换为真实语音识别。
   103→
   104→**决策：** 选用 OpenAI Whisper base 模型，本地 CPU 推理。
   105→
   106→**交付物：**
   107→- [x] Whisper base 模型懒加载（`_get_whisper_model` 单例）
   108→- [x] Opus → tmpfile → Whisper transcribe（`asyncio.to_thread` 非阻塞）
   109→- [x] 固定 `language="zh"` 中文识别
   110→- [x] 测试：mock Whisper 模型验证临时文件创建、语言参数、文本 strip
   111→
   112→---
   113→
   114→### Phase 3 — ~~OpenClaw 智能体对接~~ ✅ 已在 Phase 0.5 完成
   115→
   116→已通过 `openclaw agent` CLI 完成对接，无需额外工作。
   117→后续可考虑升级为 channel 插件获得更紧密集成。
   118→
   119→---
   120→
   121→### Phase 4 — 真实 TTS 接入 ✅ 已完成（2026-03-16）
   122→
   123→**目标：** 将 `process_tts` 从 mock 替换为真实语音合成。
   124→
   125→**决策：** 选用 edge-tts（微软 Neural TTS），MP3 → ffmpeg → Opus 编码。
   126→
   127→**交付物：**
   128→- [x] edge-tts `Communicate.stream()` 异步收集 MP3 音频
   129→- [x] ffmpeg 管道转码 MP3 → Opus（libopus, 24kbps, 16kHz, mono）
   130→- [x] ffmpeg 失败时抛出 RuntimeError
   131→- [x] 测试：mock edge-tts stream + ffmpeg subprocess，验证 MP3 拼接和错误处理
   132→- [x] 11 个测试全部通过
   133→
   134→> **详细记录见：** `MILESTONE-STT-TTS.md`
   135→
   136→**遗留（留给 Phase 5）：**
   137→- [ ] 实现分块推流：将长音频切成 ≤960 字节 Opus 帧逐帧下发
   138→- [ ] 推送 `{"type": "state", "status": "speaking"}` 状态通知
   139→
   140→---
   141→
   142→### Phase 5 — 生产加固
   143→
   144→**目标：** 具备长期稳定运行的能力。
   145→
   146→**任务：**
   147→- [ ] 连接管理：最大并发数限制、心跳超时断开、异常重连
   148→- [ ] 配置外置：`config.yaml` 或环境变量管理端口、模型选择、API 密钥等
   149→- [ ] 结构化日志：JSON 格式日志，接入日志收集
   150→- [ ] 指标监控：`/metrics` 端点暴露连接数、管线延迟、错误率
   151→- [ ] systemd 服务单元文件，支持开机自启和自动重启
   152→- [ ] HTTPS/WSS 支持（通过 nginx 反代或 uvicorn SSL 参数）
   153→- [ ] HTTP 管理接口：`GET /sessions` 查看在线连接、`POST /broadcast` 广播消息
   154→
   155→---
   156→
   157→## 技术栈总览
   158→
   159→| 层 | 当前 | 目标 |
   160→|----|------|------|
   161→| 框架 | FastAPI + uvicorn | 不变 |
   162→| WS 协议 | Starlette WebSocket | 不变 |
   163→| 数据校验 | Pydantic v2 | 不变 |
   164→| STT | Whisper base (本地 CPU) | 不变（可选升级 small/medium） |
   165→| LLM | `openclaw agent` CLI → Gateway :18789 | 同左（可选升级为 channel 插件） |
   166→| TTS | edge-tts → ffmpeg → Opus | 不变（可选 CosyVoice） |
   167→| 音频编解码 | ffmpeg (libopus) | 不变 |
   168→| 部署 | `python main.py` | systemd + nginx |
   169→
   170→## 文件结构规划
   171→
   172→```
   173→openclaw-proxy/
   174→├── main.py              # FastAPI 应用入口、WS 端点、帧路由
   175→├── pipeline/
   176→│   ├── __init__.py
   177→│   ├── stt.py           # process_stt 实现
   178→│   ├── llm.py           # call_openclaw 实现
   179→│   └── tts.py           # process_tts 实现
   180→├── models.py            # Pydantic 模型集中定义
   181→├── session.py           # 连接会话状态管理、音频缓冲
   182→├── config.py            # 配置加载（环境变量 / yaml）
   183→├── tests/
   184→│   ├── test_main.py
   185→│   ├── test_stt.py
   186→│   ├── test_llm.py
   187→│   └── test_tts.py
   188→├── requirements.txt
   189→├── CLAUDE.md
   190→└── PLAN.md
   191→```
   192→
   193→> 注：Phase 0 阶段全部逻辑集中在 `main.py`，从 Phase 1 开始按上述结构拆分模块。
   194→


Now let me examine the existing test client and requirements file, plus check if there is a MILESTONE doc with additional context.


     1→"""Quick WebSocket client to test the full proxy pipeline."""
     2→
     3→import asyncio
     4→import json
     5→import websockets
     6→
     7→
     8→async def main() -> None:
     9→    uri = "ws://localhost:8000/chat"
    10→    async with websockets.connect(uri) as ws:
    11→        print("=== Connected ===\n")
    12→
    13→        # 1) Test text frame: wake_word
    14→        print("[TX] wake_word")
    15→        await ws.send(json.dumps({"type": "hardware_event", "action": "wake_word"}))
    16→        resp = json.loads(await ws.recv())
    17→        print(f"[RX] {resp}\n")
    18→
    19→        # 2) Test binary frame: triggers STT → LLM → TTS pipeline
    20→        print("[TX] binary audio (80 bytes of mock Opus)")
    21→        await ws.send(b"\x00" * 80)
    22→
    23→        # Collect all responses until we get binary audio back
    24→        while True:
    25→            msg = await ws.recv()
    26→            if isinstance(msg, bytes):
    27→                print(f"[RX] binary audio: {len(msg)} bytes")
    28→                break
    29→            else:
    30→                data = json.loads(msg)
    31→                print(f"[RX] {data}")
    32→
    33→        print("\n=== Done ===")
    34→
    35→
    36→if __name__ == "__main__":
    37→    asyncio.run(main())
    38→


     1→fastapi>=0.115.0
     2→uvicorn[standard]>=0.34.0
     3→websockets>=14.0
     4→pydantic>=2.0
     5→openai-whisper>=20250625
     6→edge-tts>=7.0
     7→pytest>=8.0
     8→httpx>=0.28.0
     9→pytest-asyncio>=0.25.0
    10→


     1→# Milestone: Real STT + TTS Integration
     2→
     3→> **日期：** 2026-03-16
     4→> **阶段：** Phase 2 + Phase 4（合并完成）
     5→> **状态：** ✅ 单元测试全部通过，等待实机验证
     6→
     7→---
     8→
     9→## 1. 概述
    10→
    11→本次 milestone 将 openclaw-proxy 的 STT（语音识别）和 TTS（语音合成）从 mock 替换为真实实现，打通完整的语音链路：
    12→
    13→```
    14→ESP32 麦克风 → Opus 音频 → [STT: Whisper] → 文字
    15→    → [LLM: OpenClaw CLI] → 回复文字
    16→    → [TTS: edge-tts + ffmpeg] → Opus 音频 → ESP32 扬声器
    17→```
    18→
    19→**之前（Phase 0.5）：** STT 固定返回 `"测试语音输入"`，TTS 固定返回 159 字节静音帧。
    20→**现在：** STT 使用 Whisper base 模型做真实中文语音识别，TTS 使用 edge-tts 合成语音并通过 ffmpeg 编码为 Opus。
    21→
    22→---
    23→
    24→## 2. 技术选型与理由
    25→
    26→### 2.1 STT — OpenAI Whisper (base)
    27→
    28→| 项目 | 选择 |
    29→|------|------|
    30→| 引擎 | `openai-whisper` 20250625 |
    31→| 模型 | `base`（74M 参数，~140MB 磁盘） |
    32→| 运行方式 | 本地 CPU 推理（无 GPU） |
    33→| 语言 | 固定 `zh`（中文） |
    34→
    35→**为什么选 Whisper base：**
    36→- 4GB 内存 + 4 核 CPU 环境下，`base` 是精度和速度的最佳平衡点
    37→- `tiny` 太不准确，`small` 在 CPU 上太慢（>10s/句）
    38→- 本地运行，无需外部 API，无网络延迟
    39→- Whisper 内置 ffmpeg 调用，能直接处理 Opus 格式输入
    40→
    41→**不选 FunASR / SenseVoice 的原因：**
    42→- 依赖更重，安装复杂
    43→- Whisper 已满足演示需求，后续可替换
    44→
    45→### 2.2 TTS — edge-tts + ffmpeg
    46→
    47→| 项目 | 选择 |
    48→|------|------|
    49→| 合成引擎 | `edge-tts` 7.2.7（微软 Edge 在线 TTS） |
    50→| 声音 | `zh-CN-XiaoxiaoNeural`（女声，自然度高） |


Let me also check the `.claude` directory for any relevant memory files, and understand the existing test count precisely.


List .claude directory


python -m pytest tests/test_main.py --co -q 2>/dev/null || echo "Cannot list tests"

List all test names to understand exact test inventory


total 8
drwxr-xr-x 2 root root   33 Mar 16 00:23 .
drwxr-xr-x 7 root root 4096 Mar 16 00:28 ..
-rw-r--r-- 1 root root  452 Mar 16 00:23 settings.local.json


Cannot list tests


python -c "import main; print('import ok')" 2>&1 | head -5

Check if main.py can be imported (dependency availability)


     1→{
     2→  "permissions": {
     3→    "allow": [
     4→      "Bash(python:*)",
     5→      "Bash(python3:*)",
     6→      "Bash(source:*)",
     7→      "Bash(ls:*)",
     8→      "Bash(curl:*)",
     9→      "Bash(which openclaw:*)",
    10→      "Bash(openclaw gateway:*)",
    11→      "Bash(openclaw chat:*)",
    12→      "Bash(openclaw agent:*)",
    13→      "Bash(which wscat:*)",
    14→      "Bash(npm install:*)",
    15→      "Bash(which ffmpeg:*)",
    16→      "Bash(pip3 list:*)",
    17→      "Bash(ffmpeg:*)",
    18→      "Bash(tree:*)"
    19→    ]
    20→  }
    21→}
    22→


/bin/bash: line 1: python: command not found


source venv/bin/activate && python -m pytest tests/test_main.py --co -q 2>&1

List all existing test names using venv


tests/test_main.py::test_process_stt_calls_whisper
tests/test_main.py::test_call_openclaw_parses_cli_output
tests/test_main.py::test_call_openclaw_raises_on_cli_failure
tests/test_main.py::test_call_openclaw_raises_on_empty_payloads
tests/test_main.py::test_process_tts_returns_opus_bytes
tests/test_main.py::test_process_tts_raises_on_ffmpeg_failure
tests/test_main.py::test_health_endpoint
tests/test_main.py::test_websocket_connect_and_disconnect
tests/test_main.py::test_websocket_text_frame_wake_word
tests/test_main.py::test_websocket_text_frame_malformed_json
tests/test_main.py::test_websocket_binary_frame_full_pipeline

11 tests collected in 2.04s


Good. Let me now verify the exact test results and understand which tests will be directly affected by the changes.


source venv/bin/activate && python -m pytest tests/test_main.py -v 2>&1

Run all tests to confirm current green state


============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-9.0.2, pluggy-1.6.0 -- /root/projects/openclaw-proxy/venv/bin/python
cachedir: .pytest_cache
rootdir: /root/projects/openclaw-proxy
plugins: anyio-4.12.1, asyncio-1.3.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collecting ... collected 11 items

tests/test_main.py::test_process_stt_calls_whisper PASSED                [  9%]
tests/test_main.py::test_call_openclaw_parses_cli_output PASSED          [ 18%]
tests/test_main.py::test_call_openclaw_raises_on_cli_failure PASSED      [ 27%]
tests/test_main.py::test_call_openclaw_raises_on_empty_payloads PASSED   [ 36%]
tests/test_main.py::test_process_tts_returns_opus_bytes PASSED           [ 45%]
tests/test_main.py::test_process_tts_raises_on_ffmpeg_failure PASSED     [ 54%]
tests/test_main.py::test_health_endpoint PASSED                          [ 63%]
tests/test_main.py::test_websocket_connect_and_disconnect PASSED         [ 72%]
tests/test_main.py::test_websocket_text_frame_wake_word PASSED           [ 81%]
tests/test_main.py::test_websocket_text_frame_malformed_json PASSED      [ 90%]
tests/test_main.py::test_websocket_binary_frame_full_pipeline PASSED     [100%]

============================== 11 passed in 2.08s ==============================


All 11 tests pass. Now I have a complete picture. Let me trace the exact code paths that will change and verify the PLAN.md note about keeping everything in `main.py`.


Grep

Phase 1.*main\.py|Phase 1.*模块|从 Phase 1 开始


193:> 注：Phase 0 阶段全部逻辑集中在 `main.py`，从 Phase 1 开始按上述结构拆分模块。


Interesting -- PLAN.md line 193 says module splitting starts at Phase 1, but the user's design constraints say "Keep all changes in main.py (no module split yet per PLAN.md)." I will respect the user's explicit constraint and keep everything in `main.py`, deferring module splitting.

Now I have all the information I need. Let me produce the implementation plan.

---

## Phase 1 Implementation Plan: Audio Buffering / Press-to-Talk

### 1. Summary of Current State

**11 tests pass.** The codebase is a single `main.py` (309 lines) with a single test file `tests/test_main.py` (212 lines). There are no per-connection state objects. Every binary WebSocket frame independently triggers the full STT-to-LLM-to-TTS pipeline, which is the core problem this phase solves.

The existing frame router at line 193 (`handle_text_frame`) recognizes `KEY_DOWN`, `KEY_UP`, and `WAKE_WORD` via Pydantic validation, but only acts on `WAKE_WORD`. `KEY_DOWN` and `KEY_UP` are parsed then silently dropped. The binary frame handler at line 219 (`handle_binary_frame`) runs the entire pipeline on each frame.

---

### 2. Design Decisions

**Q1: Where to store SessionState?**
Decision: **Local variable in `websocket_chat`**, passed by reference to all handler functions. This is the simplest approach -- no module-level dict to worry about cleanup, no risk of stale entries after disconnect, and no concurrency issues since each WebSocket connection is a single asyncio task. The `SessionState` instance is created at the top of `websocket_chat` and passed to `handle_text_frame` and `handle_binary_frame`.

**Q2: wake_word vs key_down?**
Decision: Both trigger the `idle -> listening` transition identically. The state machine treats them as synonyms for "start recording."

**Q3: Max buffer size?**
Decision: **10 MB** (`MAX_AUDIO_BUFFER_BYTES = 10 * 1024 * 1024`). At 24kbps Opus, that is roughly 55 minutes of audio, which is generous. If the buffer exceeds this limit during `listening`, log a warning and auto-trigger the pipeline (treat it as an implicit key_up).

**Q4: Empty audio on key_up?**
Decision: Log a warning (`"key_up received but audio buffer is empty"`) and transition directly back to `idle`. Do not invoke the pipeline.

**Q5: Binary frames while idle?**
Decision: **Drop with a warning log** (strict state machine). The ESP32 firmware is expected to send `key_down` first. This prevents accidental pipeline invocations.

**Q6: Binary frames during processing?**
Decision: **Drop with a warning log.** Cannot start a new recording while the pipeline is running. The client must wait for the pipeline to finish (state returns to `idle`).

---

### 3. Step-by-Step Implementation

#### Step 0: Write failing tests first (TDD, per CLAUDE.md)

Add the following new test functions to `tests/test_main.py`. Write ALL failing tests before touching `main.py`. New tests:

1. **`test_session_state_defaults`** -- Unit test. Instantiate `SessionState()`, verify `status == "idle"` and `audio_chunks == []`.

2. **`test_session_state_validation`** -- Unit test. Verify `SessionState(status="invalid")` raises `ValidationError`.

3. **`test_key_down_starts_listening`** -- Integration test. Send `key_down` over WebSocket, verify response is `{"type": "state", "status": "listening"}`.

4. **`test_wake_word_starts_listening`** -- This already exists as `test_websocket_text_frame_wake_word`. No new test needed, but verify it still passes after refactor.

5. **`test_key_up_triggers_pipeline`** -- Integration test. Send `key_down`, then binary frames, then `key_up`. Mock `process_stt`, `call_openclaw`, `process_tts`. Verify the full response sequence: `listening` state, `thinking` state, `text_reply`, binary audio. Verify `process_stt` is called with the concatenated audio.

6. **`test_key_up_empty_buffer_skips_pipeline`** -- Integration test. Send `key_down` then immediately `key_up` with no binary frames between. Verify NO `thinking` state or pipeline output. Verify state returns to `idle` (no crash, connection stays open -- send a subsequent `wake_word` to verify connectivity).

7. **`test_binary_frame_while_idle_is_dropped`** -- Integration test. Send binary data without prior `key_down`. Verify it does not trigger the pipeline (send `wake_word` after, verify `listening` response to confirm connection is alive).

8. **`test_binary_frames_are_concatenated`** -- Integration test. Send `key_down`, then 3 separate binary frames, then `key_up`. Mock `process_stt` and verify it was called with all 3 frames joined: `b"chunk1" + b"chunk2" + b"chunk3"`.

9. **`test_repeated_key_down_resets_buffer`** -- Integration test. Send `key_down`, binary data, then another `key_down`. The second `key_down` should clear the buffer and restart. Then send `key_up`. Verify `process_stt` is called only with audio received after the second `key_down`.

10. **`test_key_up_without_key_down_is_ignored`** -- Integration test. Send `key_up` while in `idle` state. Verify no crash, no pipeline invocation. Verify connection stays alive.

11. **`test_oversized_buffer_auto_triggers_pipeline`** -- Unit test on `handle_binary_frame`. Verify that when `audio_chunks` total exceeds `MAX_AUDIO_BUFFER_BYTES`, the handler auto-triggers the pipeline. (This one may be deferred as an enhancement -- see section 5.)

Total: 8-9 new test functions (reaching approximately 19-20 tests total).

**Important existing test to update:** `test_websocket_binary_frame_full_pipeline` (line 171). Currently it sends a bare binary frame and expects the full pipeline response. After Phase 1, a bare binary frame in `idle` state will be dropped. This test must be rewritten to follow the `key_down -> binary -> key_up` sequence. This is the only breaking change to existing tests.

#### Step 1: Add `SessionState` model and `MAX_AUDIO_BUFFER_BYTES` constant

Location: `/root/projects/openclaw-proxy/main.py`, after the existing Pydantic models (after line 64).

```python
# ── Audio buffer configuration ──────────────────────────────────────
MAX_AUDIO_BUFFER_BYTES: int = 10 * 1024 * 1024  # 10 MB

class SessionState(BaseModel):
    """Per-connection state for audio buffering.

    Attributes:
        status: Current state in the press-to-talk state machine.
        audio_chunks: Buffered audio frames received during "listening".
    """

    model_config = {"arbitrary_types_allowed": True}

    status: Literal["idle", "listening", "processing"] = "idle"
    audio_chunks: list[bytes] = []

    def buffer_size(self) -> int:
        """Return total size of buffered audio in bytes."""
        return sum(len(chunk) for chunk in self.audio_chunks)

    def drain_audio(self) -> bytes:
        """Join and clear all buffered audio. Return concatenated bytes."""
        audio = b"".join(self.audio_chunks)
        self.audio_chunks.clear()
        return audio

    def reset(self) -> None:
        """Reset to idle state with empty buffer."""
        self.status = "idle"
        self.audio_chunks.clear()
```

The `Literal` import must be added to the top-level imports: `from typing import Literal`.

Note: `drain_audio` and `reset` are helper methods. They keep the state transitions centralized and testable. `buffer_size` is used for the overflow check.

#### Step 2: Refactor `handle_text_frame` to manage state transitions

Change the function signature to accept `SessionState`:

```python
async def handle_text_frame(
    ws: WebSocket, raw: str, state: SessionState, session_id: str
) -> None:
```

Add logic for each action:

- **`WAKE_WORD`**: If `state.status == "idle"`, set `state.status = "listening"`, clear `state.audio_chunks`, send `{"type": "state", "status": "listening"}`. If already `listening`, log a warning and re-send the acknowledgement (idempotent). If `processing`, log a warning and ignore.

- **`KEY_DOWN`**: Identical behavior to `WAKE_WORD`. If `idle`, transition to `listening`. If already `listening`, log warning about repeated `key_down`, clear the buffer (the user re-pressed), re-send `listening` state. If `processing`, log warning and ignore.

- **`KEY_UP`**: If `state.status == "listening"`, transition to `processing` and call `handle_pipeline`. If `idle`, log warning ("key_up without prior key_down") and ignore. If `processing`, log warning and ignore.

The `session_id` parameter is needed because `handle_text_frame` now triggers the pipeline on `KEY_UP`.

#### Step 3: Refactor `handle_binary_frame` to buffer-only

Change the function signature:

```python
async def handle_binary_frame(
    ws: WebSocket, data: bytes, state: SessionState
) -> None:
```

New logic (replaces the entire current body):

1. If `state.status != "listening"`: log warning ("binary frame received in {state.status} state, dropping") and return.
2. Check `state.buffer_size() + len(data) > MAX_AUDIO_BUFFER_BYTES`. If exceeded, log warning and return (drop the frame rather than auto-triggering pipeline -- keep it simple for Phase 1).
3. Append: `state.audio_chunks.append(data)`.
4. Log info: `"Buffered %d bytes (total: %d bytes)"`.

This function no longer calls `process_stt`, `call_openclaw`, or `process_tts`. It only appends to the buffer.

#### Step 4: Extract `handle_pipeline` from the old `handle_binary_frame`

New function containing the STT-to-LLM-to-TTS logic currently in lines 232-261:

```python
async def handle_pipeline(
    ws: WebSocket, state: SessionState, session_id: str
) -> None:
    """Run the full STT -> LLM -> TTS pipeline on buffered audio.

    Args:
        ws: Active WebSocket connection.
        state: Per-connection session state (must be in "processing" status).
        session_id: Per-connection session ID.

    The state is reset to "idle" after the pipeline completes (or on error).
    """
```

Body:
1. Assert / guard: `state.status == "processing"`.
2. `audio = state.drain_audio()`.
3. If `len(audio) == 0`: log warning, `state.reset()`, return early.
4. `try:` block containing the existing pipeline stages (STT, thinking state, heartbeat, LLM, text_reply, TTS, binary send).
5. `finally:` block: `state.reset()` -- ensures state always returns to `idle`, even on error.

The error handling in the `finally` block is important: if `process_stt` or `call_openclaw` raises an exception, the connection must not remain stuck in `processing` forever. The `finally` guarantees the state resets to `idle` so the user can try again.

#### Step 5: Update `websocket_chat` to create and pass `SessionState`

Modify the WebSocket endpoint function:

```python
@app.websocket("/chat")
async def websocket_chat(ws: WebSocket) -> None:
    """Main WebSocket endpoint for XiaoZhi ESP32 hardware."""
    await ws.accept()
    peer = f"{ws.client.host}:{ws.client.port}" if ws.client else "unknown"
    session_id = f"xiaozhi-{uuid.uuid4().hex[:8]}"
    state = SessionState()
    logger.info("Client connected: %s (session=%s)", peer, session_id)

    try:
        while True:
            message = await ws.receive()

            if message["type"] == "websocket.disconnect":
                break

            text_data = message.get("text")
            byte_data = message.get("bytes")

            if text_data is not None:
                await handle_text_frame(ws, text_data, state, session_id)
            elif byte_data is not None:
                await handle_binary_frame(ws, byte_data, state)
    except WebSocketDisconnect:
        pass
    finally:
        logger.info("Client disconnected: %s (session=%s)", peer, session_id)
```

Changes: add `state = SessionState()` after `session_id`, pass `state` and `session_id` to both handlers.

#### Step 6: Update existing test `test_websocket_binary_frame_full_pipeline`

This is the ONE existing test that must change. Currently it sends a raw binary frame and expects the pipeline response. After Phase 1, raw binary frames in `idle` state are dropped.

Rewrite to follow the press-to-talk sequence:

```python
def test_websocket_binary_frame_full_pipeline() -> None:
    """Press-to-talk: key_down -> binary -> key_up triggers STT -> LLM -> TTS."""
    client = TestClient(app)
    with (
        patch("main.process_stt", new_callable=AsyncMock, return_value="..."),
        patch("main.call_openclaw", new_callable=AsyncMock, return_value="..."),
        patch("main.process_tts", new_callable=AsyncMock, return_value=b"..."),
    ):
        with client.websocket_connect("/chat") as ws:
            # 1) key_down -> listening
            ws.send_text(json.dumps({"type": "hardware_event", "action": "key_down"}))
            resp = json.loads(ws.receive_text())
            assert resp == {"type": "state", "status": "listening"}

            # 2) Binary audio frames
            ws.send_bytes(b"\x00" * 80)

            # 3) key_up -> triggers pipeline
            ws.send_text(json.dumps({"type": "hardware_event", "action": "key_up"}))

            # 4) Expect: thinking, text_reply, binary
            msg1 = json.loads(ws.receive_text())
            assert msg1 == {"type": "state", "status": "thinking"}

            msg2 = json.loads(ws.receive_text())
            assert msg2["type"] == "text_reply"

            audio = ws.receive_bytes()
            assert isinstance(audio, bytes)
```

#### Step 7: Run all tests and verify green

Expected: all 11 existing tests pass (with the one rewrite), plus 8-9 new tests pass. Total approximately 19-20 tests.

---

### 4. Function Signature Changes Summary

| Function | Current Signature | New Signature |
|----------|------------------|---------------|
| `handle_text_frame` | `(ws, raw)` | `(ws, raw, state, session_id)` |
| `handle_binary_frame` | `(ws, data, session_id)` | `(ws, data, state)` |
| `handle_pipeline` | (does not exist) | `(ws, state, session_id)` -- NEW |
| `websocket_chat` | unchanged | unchanged (signature stays same, creates `SessionState` internally) |
| `process_stt` | unchanged | unchanged |
| `call_openclaw` | unchanged | unchanged |
| `process_tts` | unchanged | unchanged |

---

### 5. Edge Cases and Guard Rails

| Scenario | Behavior |
|----------|----------|
| Binary frame while `idle` | Drop, log warning |
| Binary frame while `processing` | Drop, log warning |
| `key_up` while `idle` | Ignore, log warning |
| `key_up` while `processing` | Ignore, log warning |
| `key_down` while `listening` (repeated) | Log warning, clear buffer, re-send `listening` ack |
| `key_down` while `processing` | Ignore, log warning |
| Empty audio on `key_up` | Skip pipeline, reset to `idle`, log warning |
| Buffer exceeds 10MB | Drop incoming frame, log warning |
| Pipeline raises exception | `finally` block resets state to `idle` |
| WebSocket disconnects mid-pipeline | `WebSocketDisconnect` exception propagates normally; `SessionState` is garbage collected with the local variable |

---

### 6. What Does NOT Change

- `process_stt`, `call_openclaw`, `process_tts` -- internals untouched.
- `HardwareAction`, `HardwareEvent`, `StateNotification`, `TextReply` -- no changes.
- `/health` endpoint -- no changes.
- WebSocket protocol messages -- no new message types added.
- `_get_whisper_model` -- no changes.
- Configuration constants -- no changes (one new constant `MAX_AUDIO_BUFFER_BYTES` added).
- 6 pipeline unit tests (`test_process_stt_*`, `test_call_openclaw_*`, `test_process_tts_*`) -- completely untouched.
- 3 integration tests (`test_health_endpoint`, `test_websocket_connect_and_disconnect`, `test_websocket_text_frame_malformed_json`) -- untouched.
- `test_websocket_text_frame_wake_word` -- should continue passing because `wake_word` still sends `listening` response.

---

### 7. Sequencing and Dependencies

The implementation order matters because of TDD:

1. **Write new tests first** (they will fail because `SessionState` does not exist and handler signatures have not changed).
2. **Add `SessionState` model and constant** to `main.py` -- makes `test_session_state_defaults` and `test_session_state_validation` pass.
3. **Refactor `handle_text_frame`** -- makes `test_key_down_starts_listening`, `test_key_up_without_key_down_is_ignored`, and `test_repeated_key_down_resets_buffer` pass.
4. **Refactor `handle_binary_frame`** -- makes `test_binary_frame_while_idle_is_dropped` pass.
5. **Extract `handle_pipeline`** + wire up `KEY_UP` in `handle_text_frame` -- makes `test_key_up_triggers_pipeline`, `test_key_up_empty_buffer_skips_pipeline`, and `test_binary_frames_are_concatenated` pass.
6. **Update `websocket_chat`** to instantiate `SessionState` and pass it to handlers.
7. **Rewrite `test_websocket_binary_frame_full_pipeline`** -- fixes the one breaking existing test.
8. **Run full test suite** -- verify all 19-20 tests pass.

Practically, steps 3-6 are tightly coupled and will be done together, but the test-writing happens first.

---

### 8. Risks

| Risk | Mitigation |
|------|-----------|
| Starlette `TestClient` synchronous WebSocket does not support interleaved send/receive well for complex sequences | Keep test sequences simple: send-all-then-receive-all within each test. The pipeline is invoked synchronously within the event loop, so responses are queued before `receive` calls. |
| `handle_pipeline` is called from within `handle_text_frame` (on `KEY_UP`). This means the WebSocket read loop is blocked during pipeline execution. | This is the existing behavior (the current `handle_binary_frame` already blocks the loop). This is acceptable for Phase 1 single-user. Phase 5 should consider `asyncio.create_task` for the pipeline. |
| `Literal` import may be missing | Add `from typing import Literal` at the top. Python 3.11 supports this natively. |
| The existing `test_websocket_text_frame_wake_word` test does not check state transitions | It only checks that `listening` is sent back. The test still passes because `wake_word` still sends `listening`. No change needed. |

---

### 9. Deferred Items (Explicitly NOT in Phase 1)

- **Module splitting** (PLAN.md mentions `session.py`, `models.py`) -- deferred per user constraint.
- **Opus frame chunking for TTS output** -- listed as Phase 5 in PLAN.md.
- **`speaking` state notification** -- listed as Phase 5 in PLAN.md.
- **Auto-trigger pipeline on buffer overflow** -- keep it simple, just drop frames. Can add auto-trigger later.
- **Concurrent pipeline execution** (`asyncio.create_task`) -- Phase 5 production hardening.

---

### Critical Files for Implementation

- `/root/projects/openclaw-proxy/main.py` - Core file to modify: add SessionState model, refactor handle_text_frame/handle_binary_frame, extract handle_pipeline, update websocket_chat
- `/root/projects/openclaw-proxy/tests/test_main.py` - Write 8-9 new test functions, rewrite 1 existing test (test_websocket_binary_frame_full_pipeline)
- `/root/projects/openclaw-proxy/PLAN.md` - Update Phase 1 checkboxes after implementation is verified
- `/root/projects/openclaw-proxy/test_client.py` - Update manual test client to follow key_down/binary/key_up sequence instead of bare binary frame