| title | Latent Capabilities |
|---|---|
| description | Features the hardware and software already support but are not yet wired up. |
- Every row below is something the hardware, voice pipeline, or brain already supports but that the current deployment doesn't use. It's raw material for the backlog.
- Organised by where the capability lives: hardware, voice pipeline, brain.
- Each row ends with a cross-reference: a related
ROADMAP.mditem, or flagged as a new-task candidate if no backlog entry exists yet. - Treat this as a menu, not a plan — some are cheap wins, others are complex.
Underlying peripherals on the M5Stack CoreS3 / StackChan kit that the firmware doesn't currently expose through MCP.
| Capability | What it unlocks | Priority | Cross-ref |
|---|---|---|---|
| 9-axis IMU shake/gesture (BMI270 + BMM150) | Tap-to-activate, shake-to-dismiss, head-tilt-aware responses | Medium | New-task candidate |
| Proximity sensor (LTR-553ALS) | Wake-on-approach; auto-dim face at idle | Low | New-task candidate |
| Ambient-light (same sensor) | Match face brightness to room lighting | Low | New-task candidate |
| NFC module | Tap an NFC-tagged toy/card to trigger a scripted interaction | Medium | New-task candidate |
| IR tx/rx | Universal remote mode (learn + replay legacy appliance codes) | Low | New-task candidate |
| microSD slot | Offline sound packs, local fallback voices, recorded memories | Medium | Partially overlaps ROADMAP.md → "Create backup script" |
| 3-zone touch panel | Multi-zone gesture controls (head-pat as a discrete event) | Low | New-task candidate |
Camera beyond take_photo |
On-device VLM preprocessing; streaming to a local vision server | Medium | Cross-refs ROADMAP.md → "Lock down for child-safe operation" (camera exposure) |
| ESP-NOW radio (Wi-Fi-band, no AP) | Peer-to-peer link to a second StackChan, or to the in-box ESP-NOW remote (see hardware.md) | Low | New-task candidate |
| In-box remote controller | M5Stack ships a handheld ESP-NOW remote with the kit. Could drive states/toggles without voice — receiver-side ESP-NOW handling would need adding to the firmware. | Low | New-task candidate |
| Hardware-enforced privacy LEDs | LED state wired to the peripheral-enable signal, not a software hint | High / safety | ROADMAP.md → "Hardwire privacy-indicator LEDs in firmware" |
| Servo velocity/acceleration caps | Calmer, safer, less-startling head motion | High / safety | ROADMAP.md → "Tame violent servo motion" |
Features xiaozhi-esp32-server supports upstream that aren't turned on or surfaced.
| Capability | What it unlocks | Priority | Cross-ref |
|---|---|---|---|
| SenseVoice Speech Emotion Recognition (SER) | Use the user's vocal emotion as LLM context (not just the LLM's own emoji output) | High | New-task candidate |
| SenseVoice Audio Event Detection (AED) | Detect bgm, applause, laughter, crying, coughing, sneezing — useful context for a kids' robot | Medium | New-task candidate |
| SenseVoice language-ID output | Detect when the user actually spoke a non-English language; respond in kind or request clarification | Low | Cross-refs the English-pin patch fun_local.py |
| Sherpa-ONNX ASR | Alternative to FunASR; fully offline, supports different languages | Low | Implemented (opt-in, #135) — SenseVoiceOnnx provider, no-torch int8 export |
| Custom wake word | Replace/add to the stock wake word via ESP-SR MultiNet | Low | New-task candidate |
| Voiceprint speaker ID | Distinguish family members; apply per-user persona/context | Medium | Cross-refs child-safety task (different guardrails for kids vs adults) |
| xiaozhi-server VLLM module | Server-side "What's in this photo?" pipeline | Medium | Already covered by the bridge-side take_photo + VLM long-poll path described in modes.md; this row tracks the upstream xiaozhi-server VLLM module, which we don't enable. |
| PowerMem | Dual-layer short-term + summarized memory (currently the dotty-pi agent owns memory via its FTS brain.db) |
Low | Would overlap with the pi agent's memory — probably don't |
Intent router (function_call mode) |
Route simple commands (turn off lights, set timer) without round-tripping to the LLM | Medium | New-task candidate |
| RagFlow knowledge base | Retrieval-augmented responses against a household doc store | Low | New-task candidate |
| Multi-device routing | Run the StackChan as one of several voice surfaces on the same pi agent brain | Low | Needs the full-module deployment (DB-backed) |
| Piper streaming synthesis | Lower first-audio latency than the current batch synthesis | Medium | ROADMAP.md → "Reduce first-audio latency" |
| ffmpeg post-processing on TTS | Robot-voice character via ring modulator / bitcrush / vocoder | Medium | ROADMAP.md → "TTS provider swap — robot-sounding voice" |
dotty-pi (pi agent + qwen3.5:4b on llama-swap) + dotty-pi-ext features that could be wired up.
| Capability | What it unlocks | Priority | Cross-ref |
|---|---|---|---|
| Streaming first-token to TTS | First-token TTS instead of waiting for the full response (perceived-latency win) | High | ROADMAP.md → "Reduce first-audio latency" |
| Long-lived pi agent sessions | Carry context across turns within a conversation without re-loading the persona each time | Medium | ROADMAP.md → "Reduce first-audio latency" |
| Tool pre-approval gate | Bridge confirms tool calls before they execute — useful for child-safety. | Medium | ROADMAP.md → "Lock down for child-safe operation" |
Wired up. The dotty-pi-ext extension exposes 5 voice tools (memory_lookup, remember, think_hard, take_photo, play_song). |
Done | — | |
| pi agent MCP-server mode | Expose the agent's tools/memory to other MCP clients | Low | New-task candidate |
Qwen3 role: "system" injection |
Move the English+emoji constraints into a proper system message instead of a prompt prefix/suffix; better MoE adherence | Medium | Rework of persona prompt structure |
| Qwen3 extended context (96K native) | Keep long conversation history / memory verbatim instead of summarising | Low | Costs more tokens per turn — probably not worth it yet |
| llama-swap latency/cost dashboard | Observability into per-turn inference cost on the local model | Low | New-task candidate |
| Model A/B for voice turns | Test a smaller/faster model for chitchat, escalate to 27B-think only when needed | Medium | ROADMAP.md → "Reduce first-audio latency" |
| Per-turn cost/trace surfacing | Expose pi agent trace data via the bridge /health or a new /stats endpoint |
Low | New-task candidate |
| pi agent cron scheduler | The robot could say "good morning" on a schedule, not just on demand | Low | New-task candidate |
None of these are feature requests — they're gaps in what we can see about the running system.
| Gap | What it'd unlock |
|---|---|
Capture a real tools/list response |
Ground-truth for the MCP tool table in hardware.md |
| Per-turn latency breakdown | Which of ASR / LLM / TTS / network is the dominant cost |
| Per-turn cost breakdown | Whether Qwen3 via OpenRouter is cheaper than a smaller local model |
| Per-session trace diff | Whether English-sandwich is still needed after a hypothetical model upgrade |
These are all feeders for the ROADMAP.md "Map the dotty-pi ↔ xiaozhi-server ↔ StackChan firmware interaction" backlog item.
| Signal | Do it sooner |
|---|---|
| Child-safety or privacy | Always |
| Reduces perceived latency | Usually |
| Uses hardware we already paid for | Often |
| Requires an external service | Often skip |
| Needs the full-module DB deployment | Bundle for a future migration |
- ROADMAP.md — live backlog; this file is a source for it, not a replacement.
- hardware.md — what the hardware features actually are.
- voice-pipeline.md — what the server supports upstream.
- brain.md — what the pi agent / Qwen / llama-swap expose.
- references.md — upstream source for every capability claim.
Last verified: 2026-05-18.