# Spoken Agent 0.5.0

Gateway 0.5.0 supports StopWatch 0.4.0 and 0.5.0. Speech is opt-in per device and
selected target; upgrading alone never enables automatic speech. No physical device
command is part of the release smoke.

## Settings and use

Long B opens Settings. A cycles the menu, B selects. Choose **AGENT SETTINGS** for
the existing model/effort controls, or **SPEECH**. Within Speech, A cycles between
**SPEECH MODE** and **READ SCREEN NOW**, B selects. In the mode picker A cycles
OFF / MANUAL / AUTO_REPLY / AUTO_SUMMARY, B saves. Hold B to close Settings.
Long A stops playback before beginning the existing push-to-talk flow.

| Mode | Automatic behavior | Explicit Read Screen / Admin Speak test |
| --- | --- | --- |
| OFF (default) | None | Allowed; explicit user action only |
| MANUAL | None | Allowed |
| AUTO_REPLY | Completed turn final reply → cleanup → TTS | Allowed |
| AUTO_SUMMARY | Completed turn final reply → DeepSeek summary → TTS | Allowed |

Settings belong to the device/target pair, not the Agent session. The small
`~/.config/esp32-device-gateway/speech.json` uses mode 0600 and atomic replacement.
`GATEWAY_SPEECH_FILE` can override the location for isolated tests. `pi:<id>` is
normalized to `agent:pi:<id>`; an unknown pair loads OFF. No database or profile
framework is used. A target change cancels pending synthesis and active playback.

Read Screen captures a plain-text description of the page beneath Settings when
Settings is opened. For the Agent page it uses the same last visible 6 tool / 8 text
lines as the renderer; the server pages use the displayed CPU, memory, disk or
network values. The device bounds the description to 1200 UTF-8 bytes, including
safe Unicode truncation. Gateway cleanup limits it to 400 codepoints. It does not
infer the visible page from the selected target.

The Admin target row has Speech Mode and Speak test. The latter always speaks
`语音播放测试成功。`, accepts only the selected target, and rejects arbitrary text.
Both POST routes are under `/api/devices/{device_id}/...` inside existing Cloudflare
Access protection. Device commands require the authenticated device WebSocket.

## Providers and limits

The first TTS provider is Cloudflare Workers AI `@cf/myshell-ai/melotts`. Configuration
uses the existing `GATEWAY_CF_ACCOUNT_ID` and `GATEWAY_CF_API_TOKEN`, plus:

```text
GATEWAY_TTS_PROVIDER=cloudflare-workers-ai
GATEWAY_TTS_MODEL=@cf/myshell-ai/melotts
```

Keep these in the existing external 0600 `gateway.env`; no second credential is
needed. The [official input schema](https://developers.cloudflare.com/workers-ai/models/melotts/schema-input.json)
requires `prompt` and accepts a `lang` string (default en, **no enum in the current
schema**). The [official output schema](https://developers.cloudflare.com/workers-ai/models/melotts/schema-output.json)
supports base64 MP3 in JSON `audio`, or binary `audio/mpeg`. REST JSON is unwrapped
from the Cloudflare `success`/`result` envelope. The provider accepts these two forms
and rejects bad status, invalid JSON/base64, empty or oversized audio.

Simple CJK detection chooses `zh`, otherwise `en`; Chinese text may include English
terms. This is verified with real mixed-language inference, not an LLM language
classification. Text has a hard 1200-codepoint bound; Read Screen uses 400,
AUTO_REPLY 600. Overlong replies stop at a complete sentence and add a screen
notice; a single sentence exceeding the budget is replaced by that notice.
Cleanup removes headings/formatting, fenced code, URLs, emoji/control characters,
collapses whitespace, shortens long paths to a filename, and redacts common
credential forms. It does not call an LLM or read any files mentioned by the text.

Compressed audio is converted with a subprocess argv (no shell):

```text
ffmpeg -hide_banner -loglevel error -nostdin -protocol_whitelist pipe
       -i pipe:0 -vn -f s16le -acodec pcm_s16le -ac 1 -ar 44100 pipe:1
```

Input/output stay in memory: compressed input ≤8 MiB; raw PCM ≤180 seconds
(15,876,000 bytes); ffmpeg deadline 15 seconds. HTTP response reads are bounded,
TTS has an overall deadline, and cancellation kills/reaps subprocesses. Only the
pipe protocol is permitted; there is no MP3 decoder on the ESP32 or audio cache.

AUTO_SUMMARY uses `haihub/deepseek-flash` already configured in Pi. Pi 0.85.1 exposes
`ModelRuntime.create`, `getModel` and `completeSimple`. The existing provider uses
a credential resolver; `integrations/pi/summary-once.mjs` calls the official model
runtime once, with no AgentSession, tools, extensions or stored conversation.
Only bounded summary text returns on stdout; credential resolution stays inside
the runtime. A literal/environment credential can also use the direct compatible
API. Provider errors and secrets are never logged or returned to a device.

Only agent, bounded session name, terminal state and this turn's final response
(≤16000 codepoints) enter the summarizer. Latest activity (≤480 characters) is used
only when the final response is empty. Prompt instructions prohibit guessed
results, commands/file lists, and following instructions inside the supplied text.
JSON must contain `status` and `speech`; allowed statuses are success, partial,
failed, needs_action. Chinese defaults to 50–120 characters (short facts are not
padded); English ≤400 characters; at most 3 sentences. A FAILED turn cannot return
success. Invalid output/provider failure produces these fixed fallbacks:

- DONE: 任务已经完成，请查看屏幕上的详细结果。
- FAILED: 任务没有完成，请查看屏幕上的错误信息。
- Other: 任务已经结束，请查看屏幕上的结果。

A DONE transport state is not by itself evidence of task success; the summary
prompt explicitly requires checking the final reply. Fallback output is intentionally
less informative. Summary failures never interrupt the Agent session.

## Final replies and deduplication

Pi 0.85.1's installed official extension types expose `agent_end.messages` for the
run, plus `agent_settled` after retries/compaction/queued continuation have finished.
The extension resets on agent_start, freezes the last assistant text at agent_end,
and emits once at agent_settled with a unique turn_id. No TUI, ANSI or session JSONL
is used for extraction. An already-running external Pi runtime must reload its
pager extension before it emits the new fields; newly resumed SDK runtimes use it
immediately. Old Pi events continue to update monitoring.

[Codex App Server](https://learn.chatgpt.com/docs/app-server) emits turn/started,
item/agentMessage/delta and turn/completed. Buffers are scoped by turn and item ID;
completed item text is authoritative, phase=final_answer takes priority, and
commentary is excluded. At completion the last final item is frozen and emitted
once. Recent history remains separate. Joining an already-running turn preserves
monitoring; a turn without its start event is not retroactively spoken.

Both paths call `on_agent_terminal(agent, session_id, state, final_text, ...)`.
Speech policy never triggers from activity, paragraph, tool or delta events.
A dedicated conversation can be an ordinary existing Pi session with AUTO_REPLY.
Recommended *manual* session prompt: answer in 2–4 spoken sentences, answer directly,
use little Markdown. The release never changes existing system prompts.

## Playback and priority

Only devices declaring `speech_playback` receive speech protocol messages. Each
connection has at most one synthesis/playback job. New requests cancel old work;
IDs and sequence numbers reject stale/missing/duplicate chunks. Gateway paces each
80 ms frame without catch-up bursts. Firmware uses an 8-frame (640 ms) FreeRTOS
queue and a separate task; the WebSocket callback validates, decodes and enqueues.
Queue overflow cancels the entire stream and sends speech_stop; no PCM is silently
dropped. A three-second missing-data timeout also releases the stream.

The existing `s_audio_mode_lock` owns ES8311/I2S0 across capture, speech and tones.
Priority: Approval > microphone recording > speech > ordinary DONE/FAILED tone.
Speech writes use small codec blocks so local cancellation is checked frequently.
Approval, long A, Settings entry, target changes and disconnect clear speech queues.
SPEAKING is an overlay and returns to the previous page after the queue drains.
It does not change the target. No new top-level gestures are added.

## External data flows

- STT: microphone WAV → Cloudflare Whisper, followed by transcript confirmation.
- TTS: cleaned text to speak → Cloudflare MeloTTS.
- AUTO_SUMMARY only: bounded Agent final reply and terminal state → Haihub DeepSeek Flash.

Full session history, source files, tool logs and credentials are not collected for
speech. Final replies can themselves contain sensitive user content: automatic
speech defaults OFF and is explicitly selected per device/target. No text/audio
payloads are exposed through the device last-event diagnostics. No wake word,
always-on microphone, full duplex, real-time model, voice cloning, profiles, OTA,
touch control, new adapters or multi-server hub is part of this change.