# Spoken Agent validation 0.5.0

Validation date: **2026-10-03 (Asia/Shanghai)**

Firmware / Protocol / Control / FastAPI: **0.5.0**.
Baseline: clean main at `6fd589a`. Release commit: the commit containing this record
(`Add spoken agent playback and summaries`). No physical ESP32 commands or flashing.

## Production

First, the accepted STT scope was recorded and Control **0.4.1** was deployed:
active, NRestarts=0, health=0.4.1, CloudflareWorkersAISttProvider loaded from the
running process environment. Exact machine identifiers are **BEST EFFORT / CONFIRM
BEFORE SEND**, not a production blocker. GATEWAY_STT_INITIAL_PROMPT remains unset.

After all backward compatibility tests passed, production was upgraded to Control
**0.5.0**. Verified: active, NRestarts=0, `/health` status=ok/control_version=0.5.0,
zero online devices; live process configuration loads Cloudflare STT and MeloTTS.
The external env file is 0600 and reuses the existing Cloudflare token.
Haihub configuration resolves through the official Pi SDK. Anonymous requests to
`/admin`, `/api/devices/synthetic/speech`, and `/api/devices/synthetic/speech-test`
returned Cloudflare Access 302 redirects. Tunnel, DNS and Access were not modified.

Physical device remains **Firmware 0.4.0** until the user flashes 0.5.0. No mode was
set on a real device. An unconfigured device/target loads OFF, and a device without
speech_playback receives no speech protocol messages.

## Real provider smoke

Reproducible entrypoint: `server/.venv/bin/python scripts/smoke-speech.py`.
Only public generated test sentences were used. Audio stayed in memory.
[Recorded public results](spoken-agent-0.5.0-smoke.json) contain no credentials,
private Agent output, original environment files, session history or audio.

Cloudflare Workers AI `@cf/myshell-ai/melotts`:

| Case | Result | PCM bytes | Duration | TTS + decode |
| --- | --- | ---: | ---: | ---: |
| Chinese | PASS | 251194 | 2.848 s | 1.765 s |
| English | PASS | 241978 | 2.744 s | 1.494 s |
| Chinese + English | PASS | 344692 | 3.908 s | 1.703 s |

All three returned successful HTTP inference, nonempty MP3, successful ffmpeg
decode, 44100 Hz / 16-bit / mono / signed little-endian PCM, and plausible duration.
The mixed sentence was `Codex 已经完成任务，pytest 全部通过。`. These are cloud/format
checks, not subjective pronunciation or physical speaker quality measurements.

Cloudflare Whisper regression: Chinese and English semantics passed. Natural
code-switching preserved the request/sequence and most terms, with the accepted
confirmation caveat: this run rendered pytest as `paytest` (and the shorter TTS
round trip as `PiTest`); checkout/feature/branch were partly joined. This is not an
exact-command transcription guarantee. Production readiness retains the user's
accepted demonstration scope; exact strings must be confirmed or set through UI.

Haihub `deepseek-flash` real one-shot summary:

| Input case | Returned status | Evaluation |
| --- | --- | --- |
| Gateway fixed, 37 tests passed, committed | success | Main result and commit preserved; no invented result |
| Dependency unavailable, no commit | failed | Did not claim success; installation next action preserved |
| Configuration fixed, unit tests passed, integration test blocked, not committed | partial | Both completed and remaining work preserved |

Every result was valid structured JSON, at most 3 sentences and below 120 Chinese
characters. Short facts were not padded. Real provider output is in the public
JSON record. Invalid JSON, provider/network error, overlong output, excess sentences,
and a success claim for FAILED are covered by fixed-fallback tests.

## Complete software chains

Reproducible entrypoint: `server/.venv/bin/python scripts/smoke-spoken-agent.py`.
It starts an isolated TLS Gateway with dedicated device credentials and temporary
projects; it never uses the production device identity or important repositories.

- Read Screen: authenticated fake WSS device → explicit screen request under OFF
  → real MeloTTS → PCM → bounded realtime fake speaker: **PASS**.
- Conversation: fake microphone PCM → explicitly **FAKE STT** → transcript
  confirmation → selected real Pi 0.85.1 temporary session → final reply →
  AUTO_REPLY → real MeloTTS → fake speaker: **PASS**, exactly one speech and zero
  summarizer calls. Real STT is independently verified above.
- Coding: real Codex App Server temporary test thread performs a safe text-only
  arithmetic check, without tools/files → final reply → AUTO_SUMMARY → real
  DeepSeek Flash → real MeloTTS → fake speaker: **PASS**, exactly one summary and
  one speech. The existing App Server daemon was reused, never restarted.

Transport: authenticated TLS WSS, **305 PCM frames**, **2,138,262 PCM bytes**,
maximum JSON frame **9,504 bytes**, fake realtime queue high-water **1/8**.
The fake device consumed at 44100 samples/second. This supports the initial paced
streaming design; no ACK sliding window was needed for this software smoke.

## Modes, extraction, UI and cancellation

OFF / MANUAL: no automatic speech; explicit reading works. AUTO_REPLY calls no
summarizer. AUTO_SUMMARY uses only the bounded final reply/state and optional empty
reply fallback. Mode store is per device/target, defaults OFF, 0600, atomic replace.
Admin selections pin device/target and refuse arbitrary Speak test text.

Pi extraction uses formal agent_end.messages and agent_settled, with a per-run
identity and one terminal emission. Codex uses per-turn/per-item buffers, final
phase selection, start/reset/delta/completion and terminal dedup. Neither uses the
12 KiB recent history as its final reply. Reconnection during a turn preserves
monitoring without speaking an incomplete historical turn.

Firmware host tests cover the shared Settings/Speech menu and long-press release
semantics, sequence validation, old IDs, mismatched end counts, and an 8-frame
queue overflow simulation. Firmware playback owns the existing codec through the
same audio lock; WebSocket code only validates/decodes/enqueues PCM.

Gateway unit and authenticated fake WebSocket integration tests cover cancellation
by replacement, target change, disconnect, Approval, microphone start and Settings;
late synthesis cancellation; stale IDs; default OFF; manual screen requests;
terminal dedup; 0.4.0 firmware with no capability. The physical scheduling and codec
sound quality still require the hardware check.

## Verification and artifacts

`./scripts/test-server.sh`: **PASS** — 127 Python tests, 17 Node tests, three C++
host executables, TCP/WebSocket control smoke, Pi/Codex authenticated TLS voice
input transport smoke. One existing Starlette/AnyIO deprecation warning remains.
`git diff --check`: **PASS**.
`./scripts/build.sh m5stack-stopwatch`: **PASS**, ESP-IDF 5.5.4.

| Artifact | Bytes | SHA256 |
| --- | ---: | --- |
| firmware.bin | 1133440 | `00984a91720beb44cab8c201bde2b370fb431d9d51aa9f91d3e60ffef55d939c` |
| merged-flash.bin | 1198976 | `8a815ad2d54f46e529a04d1da75e253037bf84c360caebc285459db1005f4a49` |

Output: `dist/m5stack-stopwatch/`. App partition: 4,194,304 bytes; free:
**3,060,864 bytes (73%)**. Bootloader: 20,832 bytes, 36% free.
Both the app descriptor and firmware hello version are 0.5.0.

```text
REAL TTS READY: YES
DEEPSEEK SUMMARY READY: YES
CONVERSATION AUTO_REPLY READY: YES (software chain; fake STT, real Pi/TTS)
CODING AUTO_SUMMARY READY: YES (real Codex/DeepSeek/TTS; fake device)
READY FOR PHYSICAL SPEAKER VERIFICATION: YES
```

Physical speaker quality is NOT verified until StopWatch 0.5.0 is flashed.