feat(learning): group workloads by repository, operation and tool

docs/handoffs/learning-grouping.md

@@ -0,0 +1,38 @@
+# Learning grouping handoff
+
+## Result
+
+Workloads now share learning within a verified git repository, grouped by operation and tool, whatever subdirectory or worktree they ran in. Each workload family has a stored project, operation and tool, with `other` as the fallback. The catalog can filter by project and operation and offers a project hierarchy that collapses small projects into one row.
+
+Measured on a private copy of real history at the final commit: profiles go from 1880 to 1433, 987 families sit in merged groups, a second preview finds nothing further, and the hierarchy has 49 rows (48 projects plus "Other projects" covering 72). A client built from `main` still previews and lists the regrouped database. The live database was only read.
+
+This change has no UI. The dashboard and native browse views are the next step.
+
+## What changed
+
+- `internal/backend/profiles_classification.go`: the `learning_classification` table, the operation and tool vocabulary, inline `sh -c` scanning, the backfill's class rules and the merge rule.
+- `internal/backend/profiles_regroup.go`: reads each metadata file into evidence, then aggregates per family, so the result cannot depend on file order. Bad records are skipped.
+- `internal/backend/profiles_similarity.go`: the observation path that classifies a job when the controller sees it.
+- `internal/backend/profiles_learning.go`, `learning.go`, `internal/remote/server.go`: project and operation on summaries, filters, hierarchy.
+- `internal/backend/scheduler_controller.go`: a failed history pass is logged and retried after ten minutes. It no longer holds admissions, which is a deliberate change from `main`. The identity cache now expires after ten minutes.
+
+## Known limits
+
+- About 1,291 of the 1,433 groups still hold one family, because 1,421 of 2,722 families have no repository that can be verified. These include deleted worktrees whose stored scope maps to no live checkout and folders outside any repository; the split between them was not measured. The browse hierarchy, not this merge, is what shortens the list.
+- Sharing raises estimates. 8 groups covering 424 families and 496 of 3,424 runs predict over the 13.6 GiB memory budget. Adaptive mode refuses to start these automatically until the variant has five successful runs of its own. Fixed and shadow modes are unaffected. The largest are the `rewrite` groups `other/shell` and `checks/shell`, at 64 to 66 GiB. Those peaks were not investigated.
+- History keeps no command line, so an older job started through `env` or `fnm` counts as `shell` or `javascript`. New jobs are classified by what the wrapper runs.
+- A path reused by an unrelated repository can assign the wrong repository through a stored scope. It only raises a conservative estimate, and an operator can ungroup. See the `shortcut:` comment.
+- A failed git probe looks like "not a repository". That checkout stays unshared until the identity cache expires and a later job is observed.
+- A permanently failing history pass shows only in the resource controller log.
+- An invalid `learning_classification` row fails catalog reads visibly. Adding tool or operation vocabulary needs a version step.
+- A family that cannot register because the registry is full still counts in the preview.
+
+## Evidence not covered
+
+- `TestRealDaemonHandoverRetainsClaimedWorkerWaitingForExternalSlot` and `TestQueueEdgeLostSubmitResponseRetainsCommittedPayload` failed once under load with "Cannot establish process identity" and passed on rerun. The message comes from `identityCommand` in `internal/backend/ownership.go`, when `ps` or `sysctl` fails or exceeds its 5-second limit. This change does not touch it, and the root cause was not diagnosed.
+
+## Next steps
+
+1. Add the project, operation and group browse hierarchy to `remote/static/profiles.js` and `native/AgentLearning.swift`, with preview fixtures and labeled before and after screenshots.
+2. After merge, the updated controller runs the pass once. Confirm the group counts with `agent-heavy profiles list --hierarchy --json`.
+3. Before switching to Adaptive mode, review the over-budget groups above.

docs/learning-grouping.md

@@ -4,29 +4,30 @@
 
 The previous matching path created one profile per strict command family and hashed almost every environment value plus HEAD into each exact variant. Agent invocation metadata and frequent commits prevented repeated measurements from accumulating. Similar tasks could share learning only after a manual merge.
 
-The new path keeps original family identities and numeric history intact, stabilizes future exact variants, and groups compatible families by a hashed repository/directory scope, tool class and operation. Borrowing keeps the complete conservative envelope. Exact successes and recent same-machine group successes are counted separately. Source revisions move to the freshness context; setup, toolchain, arguments and semantic environment remain in exact variants.
+The new path keeps original family identities and numeric history intact, stabilizes future exact variants, and groups compatible families by a hashed git repository identity, tool class and operation, independent of subdirectory. Borrowing keeps the complete conservative envelope. Exact successes and recent same-machine group successes are counted separately. Source revisions move to the freshness context; setup, toolchain, arguments and semantic environment remain in exact variants.
 
 ## Owners
 
 - `internal/backend/profiles_identity.go`: versioned stable variants and incidental environment exclusions.
-- `internal/backend/profiles_similarity.go`: bounded operation vocabulary, scope hashing, wrapper recognition, automatic membership and pinned-profile preservation.
-- `internal/backend/profiles_regroup.go`: historical metadata joins, read-only preview, revision-checked apply and the one-time controller backfill.
+- `internal/backend/profiles_classification.go`: stored classification (repository digest, tool, operation), inline program scan, wrapper unwrapping, the backfill's class rules (tool and operation from the legacy kind, then the stored executable and title; repository from the verified checkouts or from the scope stored for the family) and the shared merge planner, which never touches pinned profiles.
+- `internal/backend/profiles_similarity.go`: legacy operation vocabulary, scope hashing, repository identity and observation.
+- `internal/backend/profiles_regroup.go`: historical metadata joins, a read step that turns each metadata file into evidence and a pure aggregate over it, learning aliases from checkouts and from stored scopes, the legacy-pair conflict rule, read-only preview, revision-checked apply and the one-time controller backfill.
 - `internal/backend/profiles_store.go`: additive private schema and description observation.
 - `internal/backend/profiles_prediction.go` and `profiles_collection.go`: identical independent/batched estimates, exact counts and recent group counts.
 - `internal/backend/scheduler.go`: validates the additive group count while retaining the existing exact-run familiarity guard for waiting-helper compatibility.
 - `internal/backend/learning.go`: CLI and shared admission invalidation for corrections and retrospective grouping.
 
 ## Compatibility and rollout
 
-Identity sidecars still contain exactly the original `family`, `variant` and `context` fields. Existing jobs and their stored identities are not rewritten. New exact fingerprints use `variant-v2`; historical variants remain distinct. Additive SQLite tables retain hashed scope and fixed operation categories. Old clients can still read original measurements and group membership. Read-only clients never migrate storage.
+Identity sidecars still contain exactly the original `family`, `variant` and `context` fields. Existing jobs and their stored identities are not rewritten. New exact fingerprints use `variant-v2`; historical variants remain distinct. Additive SQLite tables retain hashed scope and fixed operation categories. The additive `learning_classification` table holds the repository digest (null when unverifiable), tool and operation; nothing is added to existing tables, and `other` or new kinds are never written to `learning_similarity`, so old clients keep reading it. Old clients can still read original measurements and group membership. Read-only clients never migrate storage.
 
 The updated controller runs its initial collection before attempting the historical pass, then rotates pending grants through the existing correction path and collects fresh estimates. Operator splits, merges, labels, resets and floors stay pinned. Active grants and explicit overrides survive. A stale preview revision cannot apply a migration.
 
-The CLI can preview and explicitly apply grouping before a full application upgrade, because the membership schema and conservative group estimates remain compatible. Future stable fingerprints and automatic grouping require the updated runtime. Replacing or publishing a signed application remains a separate delivery step.
+The CLI can preview and explicitly apply grouping before a full application upgrade, because the membership schema and conservative group estimates remain compatible. Future stable fingerprints and automatic grouping require the updated runtime. The classification backfill is gated by grouping version 2 and runs once from the controller; an explicit CLI regroup applies it but never marks it done. A failed pass is logged in the resource controller log and retried after ten minutes; it does not hold admissions, because history only improves grouping. Replacing or publishing a signed application remains a separate delivery step.
 
 ## Validation
 
-Regression coverage exercises volatile invocation metadata, source revision changes, resource-control separation, wrapper recognition, scope/operation separation, shared conservative peaks, exact versus group success counts, machine isolation, operator corrections, stale revision rejection, read-only previews, byte-for-byte numeric history preservation and migration idempotence. Batched controller predictions are compared with independent reads.
+Regression coverage exercises volatile invocation metadata, source revision changes, resource-control separation, wrapper recognition, scope/operation separation, shared conservative peaks, exact versus group success counts, machine isolation, operator corrections, stale revision rejection, read-only previews, byte-for-byte numeric history preservation and migration idempotence. Batched controller predictions are compared with independent reads. Classification coverage adds subdirectory sharing, separation by repository, operation and tool, non-git and unrecognized-tool families staying alone, families in two repositories staying alone, order-independent classification over shuffled metadata, a backfill that skips corrupt records, deleted-worktree aliasing with ambiguous scopes left unknown, the version gate (new tool or operation vocabulary needs a version step), eviction clearing the new table, old-client compatibility and catalog filters with the project hierarchy.
 
 Run the applicable Go runtime checks through the shared queue as documented in `CONTRIBUTING.md`. Preview real history with `agent-heavy profiles regroup --json`; apply only the reviewed revision as described in `docs/learning.md`. Keep live-state copies and validation reports outside version control.