Feature request: optional System One Decision Gate (von-install + /superfast) #12589

Activity

1. doudouOUC commented on Sep 24, 2026

   

   doudouOUC

   on Sep 24, 2026

   Collaborator

   More actions

   感谢这份详尽的 feature request。我逐条对照了 main (b7cd42e) 的源码，核实每一项 architectural-coherence 声明，以下是基于源码的独立评估。

   ---

   1. AUTO approval-mode classifier 作为先例 — 确认

   packages/core/src/permissions/classifier.ts 确实存在于 main HEAD。文件头注释（第 1-17 行）明确描述：

   • Two-stage flow: Stage 1 快速 shouldBlock-only (max_tokens=256, thinking off)，Stage 2 完整 { thinking, shouldBlock, reason } (max_tokens=4096)。
   • Schema-typed: STAGE1_SCHEMA (line ~128) 和 STAGE2_SCHEMA (line ~136) 用 JSON Schema 约束输出结构，additionalProperties: false，required 字段精确声明。
   • Fail-closed: 注释明确 "any non-abort failure (API error, timeout, schema failure, context overflow) returns shouldBlock=true with unavailable=true"。failUnavailable() 函数在 Stage 1 构建失败、API 调用失败时均会触发。

   报告者称其为 "a two-stage, schema-typed, fail-closed decision model" — 完全准确。将同一模式推广到 front-door 决策门在架构风格上是连贯的。

   ---

   2. qwen sandbox subcommand 作为 von-install 先例 — 确认

   packages/cli/src/commands/sandbox.ts 存在，注册为 yargs CommandModule：

   command: 'sandbox [cmd...]',
   describe: 'Inspect tool confinement, verify it, or run one confined command',

   提供 --verify (验证内核边界)、无参数 (报告当前执行策略)、-- <cmd> (在沙箱中运行单条命令) 三种模式。handler 内部动态 import sandboxConfig / execution-sandbox-settings，不做模型推理。

   这确实是 von-install 可以遵循的先例：一个新的 yargs subcommand，out-of-band 配置后端，不 bundle 模型本身。模式可行。

   ---

   3. Slash command 注册模式 — 确认

   • SlashCommand 接口定义于 packages/cli/src/ui/commands/types.ts (line 379)，包含 name、description、kind: CommandKind、action() 函数、可选 subCommands。
   • BuiltinCommandLoader.ts (line ~60+) 逐个 import 并注册所有内置 slash command（如 contextCommand、compressCommand、settingsCommand 等数十个）。
   • CommandService.ts 用 loader 模式并行加载、去重、解决命名冲突。
   • commands.ts 的 parseSlashCommand() 实现了多级 path 解析 (/config set theme dark 等)，支持 subCommands 递归匹配。

   /superfast on|off|status 可以作为一个 SlashCommand 注册，on/off/status 作为 subCommands 或 action 参数解析。config 持久化到 settings.json 的模式在该文件中已有先例（settingsCommand 等）。注册路径清晰可行。

   ---

   4. 核心：front-door / chokepoint 声明 — 需要坦诚讨论的张力

   这是本提案最关键的架构问题。我追踪了完整的 agent loop：

   调用链: client.ts:sendMessageStream() (line 3054) → Turn.run() (turn.ts:704) → LlmChat.sendMessageStream() (llm-chat.ts:2965) → 流式返回 GenerateContentResponse chunks

   Tool-use 路由的决策方式: turn.ts 第 805 行：

   const functionCalls = resp.functionCalls ?? [];
   for (const fnCall of functionCalls) {
     const event = this.handlePendingFunctionCall(fnCall);

   模型在自己的 autoregressive 输出中生成 functionCall parts — 工具调用决策是 LLM 响应的一部分，而非 harness 层的 pre-classifier 步骤。Turn.handlePendingFunctionCall() (turn.ts:893) 从模型输出的 FunctionCall 构造 ToolCallRequestInfo。

   Agent loop 继续: client.ts 在多处检查 turn.pendingToolCalls.length (line 4581, 4592, 4630, 4894, 4912) — 当模型输出了 functionCall，loop 将 tool result 作为下一次 sendMessageStream 的输入，驱动下一轮 LLM 调用。

   张力所在: 报告者提出 gate 可以 "skip expensive work on obvious requests"。但：

   • 如果工具使用决策由 LLM 的自身输出做出，那么 pre-LLM classifier 无法跳过那个 LLM 调用——正是这个调用决定了是否需要工具。
   • 要让 gate 跳过 LLM 调用本身（"can it be answered from context already in the conversation?"），gate 需要替代 LLM 做出"不需要工具"的判断并直接回答——这不再是路由优化，而是模型替代，与 "fails open / never change behavior when unsure" 的承诺矛盾。
   • 要让 gate 仅跳过工具执行（跳过 coreToolScheduler，不发起下一轮），它需要拦截 LLM 已经做出的 functionCall——此时 LLM 调用的成本已经产生了，gate 只省了 tool execution + next-turn LLM call，不是"front door"的节省。

   换言之：当前架构中不存在一个 harness 层的 pre-routing chokepoint 来决定"这个 turn 需不需要工具"——这个决策是 LLM 输出的一部分。classifier.ts 的先例是在 tool execution 之前（post-LLM-call）做安全分类，不是在 LLM 调用之前做路由分类。二者的位置不同。

   ---

   5. Shadow-mode-first 评估

   Shadow mode（分类 + 日志记录，不改变路由）是安全的——它不触及上述张力，只是旁路观测。作为第一个 increment 是合理的：

   • 不改变行为: gate 的输出只写日志，agent loop 完全不感知 gate 的存在。
   • 验证数据基础: 收集真实分布（多少 turn 是纯聊天 vs 需要工具 vs 可从上下文回答），为后续是否值得做 active gating 提供数据。
   • 基础设施先行: /v1/systemone endpoint、von-install、/superfast flag、settings 持久化都可以在此阶段交付和验证。

   但从 shadow 到 active 的路径需要重新审视: active gating 要节省 LLM 调用本身，需要 gate 做的不只是"分类"而是"替代"——即对于 gate 判定为"可从上下文回答"的 turn，直接用 System One 模型的输出回答用户。这是一个完全不同的架构（dual-model serving），而非"在现有 agent loop 前加一个 pre-classifier"。报告者应该考虑将 active 阶段的 scope 明确为：

   1. (保守) 只跳过工具执行 + 下一轮 LLM call（LLM 已发出 functionCall），节省有限；
   2. (激进) 跳过首轮 LLM call，用 System One 直接回答——需要重新定义 conversation 一致性、context 投影、history 格式，远非 "fails open" 能覆盖。

   ---

   建议

   1. Shadow-mode-first PR 适合接受——基础设施合理、不影响行为、先例充分。
   2. Active gating 的设计文档需要在 shadow 数据收集后，明确回答：gate 替代的是 LLM 的哪个决策（tool-use decision vs response generation），以及替代后的 conversation 一致性如何维持。
   3. 考虑一个中间方案：gate 不跳过 LLM call，但在 LLM 响应中注入 system reminder（类似 IDE context 注入的方式，见 client.ts:3988 的 hasPendingToolCall guard 和 context 注入逻辑），引导 LLM 在明显不需要工具的 turn 中直接回答——这不改变架构，只是 prompt-level 优化，更符合 "fails open" 的承诺。
2. Andrea-Bruno commented on Sep 24, 2026

   

   Andrea-Bruno

   on Sep 24, 2026

   Last edited by Andrea-Bruno

   ContributorAuthor

   More actions

   Thank you for such a careful, source-grounded review. Your point 4 is correct and I want to be honest about it. In the current architecture the decision to use a tool lives inside the model's own output, not in a harness step before the model, so a gate placed before the model cannot skip the call that makes that decision. You are right that the classifier.ts precedent sits after the model call, before tool execution, and that a front-door routing gate is a different position.

   This PR is deliberately only the shadow phase. It classifies each turn and writes the recommendation to the debug log as a fire-and-forget call. It never awaits the result, never blocks the turn, and never changes routing, so it does not touch the tension you described. With the feature off, which is the default, the code path is a no-op. The goal of this phase is exactly what you suggest, to collect the real distribution of turns, how many are plain chat, how many need a tool, and how many could be answered from what is already in the conversation, before we decide whether active gating is worth the cost.

   For the active phase I agree with your framing. The honest near-term win is the conservative one, where the gate skips the tool execution and the follow-up turn only when it is confident the answer is already in context, which saves less than the headline claim. Your middle suggestion is the one I find most promising and most consistent with fails open. Instead of skipping the model call, use the gate signal to add a short prompt-level hint that nudges the model to answer directly on turns that clearly need no tool. That keeps the architecture intact and only changes guidance, so a wrong gate decision degrades to the normal path rather than to a wrong answer. The aggressive path, where the small model answers the user directly and replaces the first model call, is a real dual-model serving design with its own questions about conversation consistency and history format, and we are not claiming it in this PR.

   So the plan matches your recommendation. Ship the safe shadow infrastructure now, and write the active design only after the shadow data is in, with an explicit answer to which decision the gate replaces and how consistency is held. Thanks again for taking the time to check the source line by line.
3. IRONICBo commented on Sep 24, 2026

   

   IRONICBo

   on Sep 24, 2026

   More actions

   One thing I would add to the shadow phase: version the question and ordered choices, then label the prediction against what the main turn actually did—not just whether it emitted a tool call. A no-tool turn can still be wrong or incomplete, so the useful metric is work avoided without changing task success.

   We use that boundary in Jev Social: route-level accuracy, fallback rate, and final evidence coverage are evaluated together. I maintain the project; Codex helped me verify this note.
4. Andrea-Bruno commented on Sep 25, 2026

   

   Andrea-Bruno

   on Sep 25, 2026

   ContributorAuthor

   More actions

   Thanks, that is a sharp point and I agree with it. Measuring only whether the main turn emitted a tool call is too coarse, because a turn with no tool call can still be wrong or incomplete. The metric that actually matters is work avoided without any change to task success. We will version the question and the ordered choices, then label each prediction against what the main turn really did, so the shadow phase reports avoided work only when the outcome stayed correct. That keeps the gate honest and matches the fails-open contract, and it is a better boundary than a plain tool-call check.