feat: plugin eval suite

CHANGELOG.md

@@ -6,8 +6,22 @@ project follows [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Changed
+
+- Merged `epilot-interface-designer` into `build-epilot-app`. Interface
+  design was only ever needed while building an App component, so the Volt UI
+  MCP guidance, the working method, and the `interface-principles.md` and
+  `visual-qa.md` references now live in the App skill and run as part of its
+  working method for every component that renders UI. The plugin ships six
+  skills.
+
+## [0.2.0] - 2026-09-17
+
 ### Added
 
+- Behavioral eval suite in `plugins/epilot-core/evals/` for `claude plugin eval`:
+  five routing and language cases with graders and epilot MCP mocks, plus the
+  manual checklist for other clients (moved from `tests/language-and-routing.md`).
 - Friendly display names, descriptions, and starter prompts for all six skills.
 - Consistent conversation and customer-content language guidance, with bilingual
   routing scenarios and stricter listing/MCP consistency checks.
@@ -40,6 +54,24 @@ project follows [Semantic Versioning](https://semver.org/).
 
 ### Changed
 
+- Platform discovery now tells agents to call `describe_api_operation` before
+  the first `call_api_operation` of an operation and never to guess body
+  fields or query parameters, matching the epilot MCP's request-contract view
+  of `describe_api_operation` and its service overview for
+  `search_api_operations` without a query.
+- Design guidance uses `create_journey` with `design` or the generic
+  `addDesign` / `updateDesign` operations; the epilot MCP no longer ships
+  `create_design` and `update_design` tools, nor `validate_journey_definition`
+  (the journey write tools run the same structural checks).
+
+### Fixed
+
+- The discovery reference named a `list_entity_schemas` tool that does not
+  exist; entity schemas are listed with `search_configuration` (type
+  `schema`).
+- The discovery reference recommended omitting the query on
+  `search_api_operations` to list every operation, which exceeded the tool
+  result limit.
 - Fix invalid YAML in the configuration skill description and validate all skill
   frontmatter and interface metadata in CI.
 - Display the plugin as **epilot**, keeping the `epilot-core` package identifier.
@@ -66,5 +98,6 @@ project follows [Semantic Versioning](https://semver.org/).
 - Claude Code marketplace compatibility.
 - Local App manifest inspector and repository validation tests.
 
-[Unreleased]: https://github.com/epilot-dev/agent-toolkit-for-epilot/compare/v0.1.0...HEAD
+[Unreleased]: https://github.com/epilot-dev/agent-toolkit-for-epilot/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/epilot-dev/agent-toolkit-for-epilot/compare/v0.1.0...v0.2.0
 [0.1.0]: https://github.com/epilot-dev/agent-toolkit-for-epilot/releases/tag/v0.1.0

CONTRIBUTING.md

@@ -38,7 +38,9 @@ npm run inspect:app -- /path/to/an/epilot-app
   repository, tests, screenshots, and examples.
 - Keep Lima, SAP, and other vendor-specific ERP behavior out of the core
   `epilot-core` plugin. Those integrations belong in optional plugins.
-- Add or update tests when behavior changes.
+- Add or update tests when behavior changes. Routing and language behavior
+  is covered by the eval suite in `plugins/epilot-core/evals/`; add a case
+  when you add a skill or change when a skill should trigger.
 - Keep Codex, portable Agent Plugins, and Claude metadata consistent.
 
 ## Pull requests

README.md

@@ -14,9 +14,9 @@ knowledge with live tools:
 - epilot MCP for current documentation, OpenAPI discovery, entity schemas, and
   connected-organization configuration;
 - Volt UI MCP for current components, props, and design tokens;
-- a thin platform guide plus focused skills for Apps, integrations,
-  end-to-end platform configuration, journey building, and native epilot
-  interface design;
+- a thin platform guide plus focused skills for Apps (including native
+  interface design with Volt UI), integrations, end-to-end platform
+  configuration, journey building, and workflow building;
 - a deterministic local inspector for epilot App manifests and build inputs.
 
 The toolkit intentionally excludes Lima, SAP, and other ERP-specific behavior.
@@ -131,8 +131,7 @@ only when needed:
             ├── integrate-with-epilot/
             ├── configure-epilot/
             ├── build-epilot-journey/
-            ├── build-epilot-workflow/
-            └── epilot-interface-designer/
+            └── build-epilot-workflow/
 ```
 
 ## Language support
@@ -148,8 +147,9 @@ do not add invented `en`/`de` objects to string fields or assume automatic
 listing translation. Each skill's `agents/openai.yaml` supplies its display
 name, short description, and starter prompt.
 
-Before a release, run the [language and routing checks](tests/language-and-routing.md)
-in the intended client. Structural checks do not establish model behavior.
+Before a release, run the [behavioral checks](plugins/epilot-core/evals/README.md):
+`claude plugin eval` in Claude Code and the manual script in other clients.
+Structural checks do not establish model behavior.
 
 ## Prerequisites
 

RELEASING.md

@@ -19,8 +19,9 @@ Toolkit releases use Semantic Versioning and Git tags in the form `vX.Y.Z`.
 7. Create a GitHub release using the matching changelog section.
 8. Verify installation from the GitHub marketplace in both Codex and Claude
    Code using a clean environment.
-9. Run [language and routing checks](tests/language-and-routing.md) in the
-   intended clients and record results for the release revision. Test ChatGPT
+9. Run the [behavioral checks](plugins/epilot-core/evals/README.md): the
+   automated eval suite in Claude Code and the manual script in the other
+   intended clients. Record results for the release revision. Test ChatGPT
    separately when distributing there; local installation is not publication.
 
 For a public OpenAI directory submission, verify the current listing limits

package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-toolkit-for-epilot",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Agent plugins for building Apps and integrations on the epilot platform.",
   "private": true,
   "license": "Apache-2.0",

plugins/epilot-core/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "epilot-core",
   "displayName": "epilot",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Understand the epilot platform, build Apps and integrations, and configure epilot end to end using current APIs, organization context, and Volt UI guidance.",
   "author": {
     "name": "epilot"

plugins/epilot-core/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "epilot-core",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Understand the epilot platform, build Apps and integrations, and configure epilot end to end using current APIs, organization context, and Volt UI guidance.",
   "author": {
     "name": "epilot",

plugins/epilot-core/README.md

@@ -10,7 +10,9 @@ systems to epilot. The visible name is **epilot**; the package ID remains
   whether a solution belongs in platform configuration, an App, or an
   integration.
 - `build-epilot-app` chooses the correct App surface, scaffolds with the
-  epilot CLI, and guides local development, validation, and delivery.
+  epilot CLI, designs the interface with Volt UI components and tokens so it
+  feels native and accessible, and guides local development, validation, and
+  delivery.
 - `integrate-with-epilot` connects external services directly through epilot
   APIs, webhooks, events, mappings, and synchronization patterns without
   requiring an App or assuming a particular ERP.
@@ -19,8 +21,6 @@ systems to epilot. The visible name is **epilot**; the package ID remains
   permissions, using the connected organization's actual state.
 - `build-epilot-journey` creates and updates customer-facing forms and funnels.
 - `build-epilot-workflow` creates internal processes with phases, branches, loops, and automated steps.
-- `epilot-interface-designer` uses Volt UI components and tokens to make App
-  interfaces feel native, accessible, and task-oriented.
 
 ## Tools
 

plugins/epilot-core/evals/.gitignore

@@ -0,0 +1 @@
+results/

plugins/epilot-core/evals/README.md

@@ -0,0 +1,93 @@
+# Behavioral checks for epilot-core
+
+This directory holds the plugin's behavioral test suite: realistic prompts,
+the expected routing and language behavior, and graders that score what the
+model produced. It complements `npm run check`, which validates packaging
+only. A passing manifest validator is not evidence that a model selected the
+right skill or preserved the customer's locale.
+
+The cases are client-neutral. Claude Code runs them automatically with
+`claude plugin eval`; for Codex and ChatGPT, run the same prompts by hand
+using the table below. No client loads this directory into model context.
+
+## Run automatically (Claude Code)
+
+Requires Claude Code 2.1.269 or later. Every run is a real model call on your
+account. Run from the plugin root:
+
+```bash
+cd plugins/epilot-core
+claude plugin eval . --no-publish --threshold 0.8 --max-cost-usd 20
+```
+
+Each case runs three times with the plugin and three times without it, so the
+summary shows what the plugin contributed (`Δ`). A full run of the current
+suite is about 30 agent runs and roughly 6 USD.
+
+Useful variants:
+
+```bash
+# one case, one run, no baseline: cheap iteration on a grader or description
+claude plugin eval . --case routes-custom-widget-to-app --runs 1 --ablation none
+
+# keep each run's transcript for inspection
+claude plugin eval . --keep-temp
+```
+
+Results land in `results/<timestamp>/` with `aggregate-result.json` and a
+self-contained `report.html`. The `results/` directory is ignored by git.
+
+### Mocks
+
+`mocks/epilot/` answers the epilot MCP tools `search_docs`, `fetch_doc`,
+`search_configuration`, and `whoami` with fixed text, so runs never contact a
+live organization. The Volt UI server has no mock and is not started. Pass
+`--allow-real-servers` only when you deliberately want live tool access.
+
+### Reading the result
+
+- `tool_used: Skill` graders are indicators, not scored, in a two-arm run.
+  They tell you whether the plugin's skill fired on natural phrasing.
+- A `Δ` near zero with the skill indicator passing means the model already
+  handles the prompt well; the plugin's value is consistency and routing.
+- A failing skill indicator is usually a `description` problem in the skill's
+  `SKILL.md`. Adjust the description and re-run the same case.
+
+## Cases
+
+| Case | Prompt (fixture) | Expected behavior |
+| --- | --- | --- |
+| `routes-business-outcome-to-configuration` | "Customers should be able to request a heat pump inspection on our website, and every new request should automatically create a task for our sales team. What do we set up in epilot for this? Advice only." | Uses `configure-epilot` or `epilot-platform-guide`; proposes a journey plus an automation or workflow; no App; no writes. |
+| `routes-custom-widget-to-app` | "Ich brauche ein eigenes Widget auf einer epilot-Entität, das Daten aus unserem internen System anzeigt. Skizziere die Umsetzung, ohne Dateien zu ändern." | Uses `build-epilot-app`; answers in German; frames the solution as an App (manifest, entity surface, CLI deployment); creates no files. |
+| `routes-erp-sync-to-integration` | "Our ERP needs to receive every new order created in epilot and later push the invoice status back onto that order. We don't want to build an epilot App. How should we wire this?" | Uses `integrate-with-epilot`; outbound and inbound mechanism; mentions idempotency, retries, or a correlation key; no App. |
+| `journey-copy-keeps-sie-form` | "Entwirf eine neue deutsche Bestätigungsseite im Stil unserer Journey. Nur Text, nichts speichern." Existing copy: "Bitte geben Sie Ihre Kontaktdaten ein." | Uses journey guidance; German copy consistently in Sie form; no invented response-time promise; no writes. |
+| `ignores-unrelated-translation` | "Übersetze: The meeting starts tomorrow." | Translates directly without activating any epilot skill or tool. |
+
+## Manual checks for other clients
+
+Run these in fresh chats in every client targeted for release, using the
+plugin's installed copy. Provide only the prompt and stated fixture; keep the
+expected behavior out of the model's input. Record the client and version,
+model, plugin revision, selected skills, tool calls, output, and pass/fail.
+Use fixtures or a test organization, with no live mutations. Evaluate the
+observable outcome, not exact wording.
+
+Run the five cases above, plus these language cases that are not yet
+automated:
+
+| Case | Prompt and fixture | Expected behavior |
+| --- | --- | --- |
+| German routing | "Kunden sollen online einen Netzanschluss beantragen. Welche Teile davon können wir direkt in epilot konfigurieren? Bitte nur beraten." | Responds in German; uses the platform guide and configuration guidance as needed; distinguishes native configuration from an App or integration; performs no writes. |
+| Different output language | "Erstelle drei englische Labels für einen neuen Tarifvergleich. Erkläre deine Auswahl auf Deutsch. Nur Entwurf." | English labels and German explanation; keeps the two language requirements separate. |
+| Preserve existing locale | "Improve the wording of this journey; keep its current language and tone. Draft only." Fixture: locale `de-DE`, heading "Dein Tarif", label "Deine E-Mail-Adresse". | German customer copy in du form, even though the request is English; no whole-resource translation and no writes. |
+| Preserve identifiers | "Erkläre diese Zuordnung auf Deutsch, ohne sie zu ändern: customer_number -> customer_number; status = active; locale = de-DE." | German explanation; all supplied identifiers and enum values remain byte-for-byte unchanged. |
+| Preserve translations | "Ergänze den deutschen Anzeigenamen, ohne andere Sprachen zu verändern. Nur JSON ausgeben." Fixture: `{"name":{"en":"Customer portal","fr":"Portail client"}}`. | Adds `name.de` with a German label and retains the existing keys and values; does not turn the resource into a plugin manifest. |
+
+## Adding a case
+
+Create `evals/<case-name>/prompt.md` with the prompt as a user would type it,
+never naming the skill, and one grader per file under `graders/`. Pair one
+grader on the result (`llm` rubric or `regex`) with one on how the model got
+there (`tool_used: Skill`). Prefer `regex` for long output and short concrete
+PASS/FAIL rubrics for `llm` graders. Add the case to the table above so the
+manual script for other clients stays complete.

plugins/epilot-core/evals/ignores-unrelated-translation/graders/german-translation.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: 'morgen'
+flags: i
+target: last_message
+---

plugins/epilot-core/evals/ignores-unrelated-translation/graders/no-skill-fired.md

@@ -0,0 +1,7 @@
+---
+type: tool_used
+tool: Skill
+min: 0
+max: 0
+arm: both
+---

plugins/epilot-core/evals/ignores-unrelated-translation/prompt.md

@@ -0,0 +1,8 @@
+---
+description: A plain translation request must be answered directly without activating any epilot skill.
+tags: [routing, negative]
+max_turns: 5
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Übersetze: The meeting starts tomorrow.

plugins/epilot-core/evals/journey-copy-keeps-sie-form/graders/no-invented-promises.md

@@ -0,0 +1,8 @@
+---
+type: llm
+---
+
+Judge only the customer-facing confirmation-page copy, not the author's notes or explanations around it.
+
+PASS if the copy is in German, addresses the customer consistently in the formal Sie form (Sie, Ihre, Ihnen), and does not state a concrete response time, deadline, or guarantee that the request did not provide (such as "innerhalb von 24 Stunden" or "in 2 Werktagen"), even if it is labelled as a placeholder. Neutral placeholders like {{Reaktionszeit}} are acceptable.
+FAIL if the copy uses the informal du form (du, dein, dir), is not in German, or includes a specific response-time or delivery promise.

plugins/epilot-core/evals/journey-copy-keeps-sie-form/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:[\w-]+:)?(?:build-epilot-journey|configure-epilot)"'
+---

plugins/epilot-core/evals/journey-copy-keeps-sie-form/graders/uses-sie-form.md

@@ -0,0 +1,5 @@
+---
+type: regex
+pattern: '\b(Sie|Ihre|Ihr|Ihnen|Ihren|Ihrer)\b'
+target: last_message
+---

plugins/epilot-core/evals/journey-copy-keeps-sie-form/prompt.md

@@ -0,0 +1,10 @@
+---
+description: Drafting German journey copy must preserve the existing formal Sie register and not invent promises.
+tags: [language]
+max_turns: 15
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Entwirf eine neue deutsche Bestätigungsseite im Stil unserer Journey. Nur Text, nichts speichern.
+
+Bestehender Text in der Journey: "Bitte geben Sie Ihre Kontaktdaten ein."

plugins/epilot-core/evals/mocks/epilot/fetch_doc.md

@@ -0,0 +1,7 @@
+---
+type: fixed
+---
+
+# {{input.path}}
+
+This document describes the requested epilot concept. Configuration is layered: data model (schemas, attributes, relations) → capture (journeys, mappings) → process (automations, workflows) → access (portals, roles). Custom code belongs in an App only when supported configuration cannot deliver the outcome. External systems connect through public APIs, webhooks, events, and Integration Toolkit mappings.

plugins/epilot-core/evals/mocks/epilot/search_configuration.md

@@ -0,0 +1,8 @@
+---
+type: fixed
+---
+
+index_status: ready
+results: []
+
+No matching configuration found in the connected organization for "{{input.query}}". The organization has the default schemas (contact, account, opportunity, order, product, price) and no custom journeys, automations, or workflows yet.

plugins/epilot-core/evals/mocks/epilot/search_docs.md

@@ -0,0 +1,11 @@
+---
+type: fixed
+---
+
+Results for "{{input.query}}":
+
+1. Journeys — customer-facing multi-step forms built in the Journey Builder. A journey submission creates or updates entities (contact, opportunity, order) through an entity mapping.
+2. Workflows (Prozesse) — internal task graphs with phases, decision branches and automation tasks; started manually, by an automation, or from a journey submission.
+3. Automations — event-driven flows: trigger (entity created/updated, journey submitted) → actions (create task, send email, start workflow, call webhook).
+4. Apps — custom code surfaces: entity tabs/widgets, custom journey blocks, portal widgets, flow actions, scheduled functions. Declared in manifest.json, deployed with the epilot CLI.
+5. Integration Toolkit — inbound/outbound mappings and webhooks for exchanging data with external systems without an App.

plugins/epilot-core/evals/mocks/epilot/whoami.md

@@ -0,0 +1,5 @@
+---
+type: fixed
+---
+
+organization: "Eval Org (test)"  org_id: "eval-org"  access: read-only

plugins/epilot-core/evals/routes-business-outcome-to-configuration/graders/native-configuration-plan.md

@@ -0,0 +1,6 @@
+---
+type: llm
+---
+
+PASS if the reply recommends native epilot configuration for this outcome, naming at least a journey (or equivalent customer-facing form) for the request AND an automation or workflow that creates the sales task, and does not recommend building a custom App or writing custom code as the primary solution.
+FAIL if the reply proposes custom code, an App, or an external integration as the main approach, or if it only describes epilot in general terms without a concrete setup plan.

plugins/epilot-core/evals/routes-business-outcome-to-configuration/graders/no-writes.md

@@ -0,0 +1,6 @@
+---
+type: llm
+---
+
+PASS if the reply stays advisory: it explains what to configure and does not claim to have created, changed, or deployed anything in an organization.
+FAIL if the reply states that it created, updated, or deployed any journey, automation, workflow, schema, or other resource.

plugins/epilot-core/evals/routes-business-outcome-to-configuration/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:[\w-]+:)?(?:configure-epilot|epilot-platform-guide)"'
+---

plugins/epilot-core/evals/routes-business-outcome-to-configuration/prompt.md

@@ -0,0 +1,8 @@
+---
+description: A plain-language business outcome should be routed to native epilot configuration, not custom code.
+tags: [routing, smoke]
+max_turns: 15
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Customers should be able to request a heat pump inspection on our website, and every new request should automatically create a task for our sales team. What do we set up in epilot for this? Advice only, please do not create or change anything.

plugins/epilot-core/evals/routes-custom-widget-to-app/graders/answers-in-german.md

@@ -0,0 +1,6 @@
+---
+type: llm
+---
+
+PASS if the reply is written in German (code identifiers, file names, and product names may stay in English).
+FAIL if the main explanatory prose is in English or another non-German language.

plugins/epilot-core/evals/routes-custom-widget-to-app/graders/app-approach.md

@@ -0,0 +1,6 @@
+---
+type: llm
+---
+
+PASS if the reply frames the solution as an epilot App: it mentions an App manifest or component, the entity surface (tab, widget, or capability on an entity), and the epilot CLI or deployment/installation step, and it presents this as a plan without claiming files were created or changed.
+FAIL if the reply suggests this can be done purely with native configuration, omits the App concept entirely, or claims to have created or modified files.

plugins/epilot-core/evals/routes-custom-widget-to-app/graders/no-files-created.md

@@ -0,0 +1,5 @@
+---
+type: file_exists
+path: "**/*"
+exists: false
+---