A coordination layer, not a compute farm. Agents bring their own compute and submit signed expert adapters to a canonical head per domain — a router + library of experts (MoE commons), never a monolithic mergeable blob. ARTISAN verifies signatures, scans compliance, independently re-evaluates every submission on its own harness (self-reported scores are recorded for audit and otherwise ignored), and promotes only what measurably beats the current head. Lineage is append-only and hash-chained; rollback is a first-class event, never a rewrite.
The rule: the vote decides what gets IN; the benchmark decides what is BETTER. This MVP implements the benchmark half end-to-end; governance/voting is the next layer.
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python demo/prove_the_loop.py # the key deliverable
.venv/bin/python demo/api_smoke.py # same loop over HTTPprove_the_loop.py seeds two agents (Corban + Ashvale) on a toy, low-IP
text-transform domain. Each fits an expert on its own data, signs a
manifest, and submits. You will see: an honest promotion (with an inflated
claimed score ignored), a fraudulent submission rejected by re-eval and
slashed, a dirty dataset manifest blocked by compliance, the head climbing
0.00 → 0.25 → 0.50 → 0.75 with zero centralized training, a rollback +
recovery, and the full hash-chained lineage verified.
Run the API standalone:
ARTISAN_ROOT=var/api .venv/bin/uvicorn api.app:app --reload
# then hit http://127.0.0.1:8000/docsThe council's verdict on the MVP: the governance skeleton is real, but the
toy proof sidestepped the one hard problem (the router) and the "zero
central training / storage-only cost" claim was too strong. plane/ is
the answer — the same loop over a REAL pinned model, REAL safetensors
LoRA experts that OVERLAP, and a REAL router refit under a hard budget
with its GPU-seconds and dollars METERED.
python3.11 -m venv .venv-plane && .venv-plane/bin/pip install -r requirements-plane.txt
.venv-plane/bin/python -m plane.run_experiment --tiny # CPU mechanics dry-run
.venv-plane/bin/python -m plane.run_experiment # real run (GPU)
python3 plane/runpod_run.py --dry-run # RunPod plan (no spend)
python3 plane/runpod_run.py # BILLABLE: provision->run->terminateWhat it implements (each a council-flagged fix):
| Council flaw | Fix in plane/ |
|---|---|
| base drift invalidates the expert library | basepin.py — base pinned by repo+revision+file sha256s+arch descriptor; adapters citing another fingerprint are rejected |
| adapters could be hostile code | abi.py/abi_sandbox.py — tensors-only safetensors, strict ABI (rank/targets/dtype), validated in an rlimited subprocess (no torch, no pickle) before touching the serving process or GPU |
| the router was dodged by orthogonal tasks | router.py — logistic gate over frozen-base embeddings, arms = base / each expert / blend, experts OVERLAP by construction (domain.py), refit on every submission under a HARD wall-clock budget |
| "zero central training" contradiction | router refit + sealed evals are metered per phase in GPU-seconds and $ (meter.py); the report states $/promotion instead of denying the cost |
| benchmark hill-climbing (score oracle) | evalplane.py — private-secret rotating shards, submitters receive coarse {status, band} only; untouched audit shard scored at report time |
| fixed +0.02 margin isn't statistics | paired bootstrap: promote only if the one-sided 95% lower bound of the per-item delta clears 0, AND no capability regresses significantly |
| concurrent evals fork lineage | Registry.insert_head_cas — atomic compare-and-swap on the head version |
| Ed25519 = free Sybil identities | registration bond + eval fee charged BEFORE any GPU spins; fraud slashes the bond (ledger units here; real money/PoW in production) |
run_experiment.py runs the whole drama end to end: genesis on the
pinned base, two honest submitters training overlapping experts on their
own compute, an adversary (NaN adapter, wrong-rank adapter, noise
adapters with inflated claims) ground down by sandbox + gates + stake,
then an audit-shard report, per-arm diagnostics (does the routed head
beat every single expert alone?), and the full meter dump.
The loop was executed on a real GPU against the hash-pinned
Qwen/Qwen2.5-0.5B-Instruct (fingerprint 5fd354f5…, rev
7ae55760…). What happened, with real numbers:
- Genesis base-only sealed pass rate 0.0000 (the aggregate metric is strict full-item exact match; the base already handles some fields — audit per-cap date 0.926, money 0.767, email 0.638, phone 0.248 — but name 0.000 formatting sinks every multi-field item).
- Both honest experts trained cleanly (loss ≈ 0.001 / 0.007, ~95 GPU-s each on their own compute) and each raised the candidate aggregate 0.00 → 0.20 — the adapters genuinely add signal.
- Both were REJECTED by the per-capability regression gate, not the aggregate: expert A (date/money/name) regressed email −0.109; expert B (email/phone/name) regressed date −0.160, money −0.155. The budget-refit router mis-routes overlapping items to the wrong expert, and the regression gate correctly refuses a net-positive-looking candidate that harms a capability. This is the council's "expert interference" made concrete and caught by governance — the exact failure the toy proof had hidden by making tasks orthogonal.
- Sandbox rejected the NaN adapter (non-finite) and the wrong-rank adapter (shape mismatch); the noise adapter was gate-rejected + stake- slashed, and the 4th submission was blocked pre-GPU with "insufficient stake" — Sybil economics working before any compute burns.
- Lineage hash chain verified; the head correctly stayed at v1 with 0 experts (governance promoted nothing false).
Metered cost (the number the council demanded): ARTISAN's central compute for one full cycle — genesis eval + two budgeted router refits
- four paired sealed evals + audit — was 452 GPU-seconds = $0.058 at
the pod's real $0.46/hr; total pod lifetime $0.098. Submitter-side
LoRA training (~189 GPU-s) costs ARTISAN $0. Projected $/promotion
≈ $0.03–0.06 of ARTISAN GPU for a candidate that clears the gate (one
refit + one paired eval on a 0.5B). That is real, small, and mostly
pushable onto submitters — not "storage-only", exactly as the council
corrected. The economics hold; the router quality (not the bill) is the
open problem this run surfaced. See
var/plane_real_runpod/RUN1_authoritative_results.md.
The v1 logistic router mis-routed overlapping items and got every honest
expert rejected. router.py's SoftRouter applies the Sovereigns's
cheap-first trio — (a) route on the frozen base's own hidden-state
embedding, (b) soft top-2 (blend when the gate spreads across both
experts), (c) a confidence fallback to base whose threshold is
calibrated on the fit pool against the per-capability no-regression
objective (reusing already-generated arms — zero extra GPU). Re-run on
a real RTX A5000, same Qwen2.5-0.5B / same overlapping experts:
- Step 0 — per-arm-ALONE diagnostics prove the adapters were never the
problem; the router was. On the audit shard each expert alone is
excellent on its own capabilities and destructive off them:
- base: date 0.974, money 0.856, name 0.000, email 0.589, phone 0.221
- expert A alone: date 1.000, money 0.982, name 0.898, email 0.056, phone 0.071
- expert B alone: email 0.976, phone 1.000, name 1.000, date 0.128, money 0.000
- uniform blend: 0.520 aggregate (date .863 money .378 name .898 email .903 phone .646)
- Expert A now PROMOTES — the fix worked. v2 routed 89/160 items to base (fallback) and 71 to expert A: aggregate delta +0.224, 95% lower bound +0.180, and zero per-capability regressions (v1 had rejected the identical adapter for email −0.109). Head advanced v1→v2, sealed score 0.000 → 0.224, audit name 0.000 → 0.551.
- The two-expert case still trips the gate — honestly. Adding expert B on top lifts aggregate even more (delta +0.268, candidate pass 0.524) but regresses date −0.147, money −0.204: with both experts present the router leaks date/money items into expert-B / blend paths that are weak there. The per-capability gate correctly blocks it — the system improved the model without ever shipping a regression.
- Guardrails all fired: sandbox blocked NaN + wrong-rank; Sybil bond/fee blocked Sly's 4th submission pre-GPU; lineage hash chain verified.
Metered: ARTISAN central compute 608 GPU-s = $0.0456 for the full cycle; $/promotion = $0.046 (1 promotion); pod lifetime $0.074 on an A5000 @ $0.27/hr; submitter LoRA training (286 GPU-s) costs ARTISAN $0.
Honest verdict: the cheap trio fixed the overlap mis-routing for the
promotable single-expert case — the exact failure that blocked v1 is
gone, and ARTISAN promoted a real 0→0.224 improvement with no regression
at ~5 cents. It did not fully solve multi-expert overlap (adding a
second overlapping expert still regresses the first's capabilities); that
needs the deeper per-field/token routing + anti-regression training loss.
Governance held throughout: it shipped the safe win and caught the unsafe
one. See var/plane_real_runpod/RUN2_v2_soft_router_stdout.log.
The frontier council reviewed v2, flagged four fixes, and gave the
architecture. plane/router.py now implements all of them:
- #1 frozen-base embedding contamination —
use_arm()(restoring context manager) +embed_base()(features always under the frozen base, finiteness-checked) + a recordedfeature_fingerprint. Active. - #2 in-sample calibration — the soft-gate now fits on a TRAIN split and calibrates τ/η on a HELD-OUT split (ε ≤ sealed ε); the field router assigns each field on TRAIN and validates no-regression OUT-OF-FOLD.
- #3 train/serve mismatch — soft KL targets + inverse-freq class weights; blend dropped from the classifier's target classes.
- #4 field-level late fusion (
FieldMergeRouter) — generate the full JSON under base + each candidate expert, then assign each capability to its best out-of-fold-validated arm (revert to base if it would regress), and merge per field. - #5 hard budget deadline in the field refit.
Real run (Qwen2.5-0.5B, RTX 3090/A5000, both overlapping experts):
- Bug #2 fix is proven honest on held-out data. Expert A's per-field no-regression, validated OUT-OF-FOLD: date calib_base 0.967 → expert 1.000 (n=31), money 0.788 → 1.000, name 0.000 → 0.818; email & phone auto-reverted to base because the expert would regress them. The "zero regression" is no longer in-sample optimism — it survives the held-out check.
- Field-merge kills the per-capability value interference the council targeted. With expert B merged in, the head's per-capability audit was name 0.00→1.00, email 0.57→0.98, phone 0.17→0.98, with date and money preserved at base — zero regression, which is exactly what whole-item routing could not achieve. Expert B promoted via field-merge in the owner-presence run (delta +0.06, 95% lb +0.036, no per-cap blocks); Expert A did not (its target caps date/money are already high in base, leaving little aggregate headroom).
- The residual bottleneck is field PRESENCE, not value. Exact-match aggregate promotion is now gated by which keys to emit — no single arm (base included) detects the item's field-set cleanly, so the merged key-set is noisy. A base-as-presence-oracle variant scored worse (base's own key-set detection is poor), confirming presence — not overlap — is the open problem. The fix is schema-constrained decoding or a presence vote, not more routing.
Metered: owner-presence field-merge run — ARTISAN central compute 701 GPU-s = $0.090, 1 promotion → $/promotion $0.090; pod lifetime $0.13 (RTX 3090 @ $0.46/hr). A cheaper A5000 ($0.27/hr) variant ran 544 GPU-s = $0.041. Submitter LoRA training costs ARTISAN $0.
Honest verdict: the council's fixes land. Bug #1 (embedding) is
enforced with a fingerprint; bug #2 (OOF calibration) is proven honest on
held-out data; field-level late fusion (#4) structurally eliminates the
per-capability value-interference that blocked v2's second expert — the
per-cap audit shows every capability at or above base. What it does not
yet deliver is a large exact-match aggregate promotion, because a newly
isolated bottleneck — field-presence detection — now dominates that
strict metric. Governance held throughout (promoted only the safe change,
caught every unsafe one, sandbox + Sybil + lineage all fired). Net: the
overlap value problem the council set out to fix is solved and honestly
validated; the next target is presence/key-set detection, not routing.
See var/plane_real_runpod/RUN_v3b_base_oracle_stdout.log.
The council caught a metric bug: v3's internal per-cap check
(_percap_accuracy) and the per-cap gate scored value-correctness only
where a field was emitted — blind to false-positive keys and recall. So
the "presence bottleneck" was partly a measurement artifact. v4 fixes the
metric and adds the missing piece:
- Corrected metric (
domain.score_full): exact key-set over the full schema — a hallucinated key (FP) and a missing key (FN) both score 0 — plus key-set precision/recall. The sealed per-cap gate now uses it. - Supervised presence head (
PresenceHead): a multilabel logistic on the frozen-base embedding, gold labels free from the fit shard, per-field thresholds calibrated out-of-fold. Gemini's hypothesis proved exactly right — the base's hidden states encode presence even though its generation does not. - Presence-masked serve: the head predicts the key-set; each present field's value comes from its OOF-validated owner arm; a presence=0 key is never emitted (kills FP); arms owning no predicted field are skipped.
Real run (Qwen2.5-0.5B, RTX 3090, both overlapping experts, corrected exact-key-set gate):
- The presence head is the breakthrough. Held-out key-set accuracy 0.891 (A) / 0.969 (A+B) vs base 0.031 — ~30× base's generation-based emission. Per-field presence F1 jumped e.g. money 0.56→1.00, phone 0.78→1.00, name 0.66→0.97.
- BOTH experts promoted honestly under the corrected metric. Expert A: aggregate delta +0.376, 95% lb +0.328, no per-cap blocks → head 0.000→0.376. Expert B on top: delta +0.472, 95% lb +0.416, candidate exact-match 0.892, no per-cap blocks → head sealed score 0.892 with 2 overlapping experts. Audit per-cap: date 1.00, money 0.99, name 0.91, email 0.94, phone 0.97 — every capability far above base, zero regression under the FP/FN-aware metric.
- Value owners OOF-validated, and reversion still fires: for A,
datereverted to base (held-out calib_expert 0.963 < base 1.000); money/name went to A. For A+B, date/money from A, name/email/phone from B — each held-out-validated ≥ base. - Did prior no-regression survive the corrected metric? Yes for the promoted head, and the metric exposed how weak base really is (base field-exact FP/FN-aware: name 0.04, money 0.35, phone 0.30) — which is why v3's tiny 0.06 "promotion" was near-noise. v4's head sits at 0.86 audit exact-match with no per-cap below base.
- Guardrails: sandbox blocked NaN + wrong-rank; Sly's noise adapters now score negative deltas under the corrected metric and are rejected; lineage verified.
Metered: ARTISAN central compute 1209 GPU-s = $0.155 for the full two-promotion cycle; $/promotion = $0.077; pod lifetime $0.19 (RTX 3090 @ $0.46/hr). Submitter LoRA training costs ARTISAN $0.
Bottom line: across v1→v4 the make-or-break question is answered
yes: real, overlapping expert-adapters plus a cheaply-refit router
make one frozen open model measurably better — 0.00 → 0.892 exact-match
from two overlapping experts, no per-capability regression, ~$0.077 per
promotion, with every governance guardrail (hash-pinned base, sandboxed
safetensors ABI, sealed rotating benchmark, confidence-bound + corrected
per-capability gate, atomic CAS, Sybil bond/fee, GPU-second metering)
intact. The council's four review rounds each found a real flaw and each
fix moved a real number. See
var/plane_real_runpod/RUN_v4_presence_stdout.log.
Per domain, ARTISAN exposes a KEEL: a fixed interface contract (inputs / outputs / controls) + a stable UI users learn once. Whole occupants — a single model, or the router+expert composite grown by the inner loop — plug into the socket and compete to hold the slot. The reigning champion serves until a challenger beats it on the existing eval referee; then the occupant is swapped atomically. The engine changes; the user's controls never do (steering wheel stays / engine swaps).
.venv/bin/python demo/prove_the_keel.py # the proof: swap behind an unchanged socket
.venv/bin/python -m keel.ui # stable dashboard at http://127.0.0.1:8777Nothing hard was rebuilt: eval/ is the swap referee, registry/ heads
- CAS + lineage record who holds the slot,
orchestrator/supplies the promotion margin and grows composite challengers,compliance/scans occupant manifests. New inkeel/: the contract (contract.py, content- addressed — the registry pins its digest on every head version, so "the interface never changed across the swap" is verified from history), thin occupant adapters (occupants.py), the referee shim that drives the existing benchmark through the socket (referee.py), the slot runtime with atomic challenge/swap/rollback (runtime.py), and the contract-generated dashboard (ui.py).
The two honest keel decisions, answered here: interface changes are
versioned (a new contract fingerprint = a new slot, never a silent
mutation), and slot-holding policy is enforceable at challenge() (the
demo is open-to-all; "only open models wear the crown" is one gate).
The complete product surface: a single-page app (landing, sockets
gallery, socket detail with a live "try it", the challenge/submission
flow, the Rank-of-Contribution board with the canonical insignia colors,
and agent profiles) wired to the real backend — the text-transform
socket runs the true compliance → referee → swap pipeline through
keel/ + registry/ + eval/ + orchestrator/, not a mockup.
.venv/bin/python -m web # http://127.0.0.1:8080 (also writes web/preview.html)
.venv/bin/python -m web --preview # only (re)generate the standalone previewweb/preview.html is a self-contained file (all CSS/JS inline, zero
network calls, sample data, an in-browser "try it") to open directly and
show the interface. Deploy split (SPA → Cloudflare Pages, API → Fly.io)
and the two things that need the Sovereigns are in web/DEPLOY.md.
The spacesuit/wings: an agent's LLM connects to the harness daemon over
MCP and operates in EuEarth through mediated tools — DID identity,
human→agent delegation credential (verified per action), capped
allowlisted session wallet (investment unrepresentable), edge
filter + C2PA-style provenance (preflight only; the server re-validates),
subprocess sandbox, and RoC-rank tool gating — all bridged to the LIVE
keel/registry/eval/web backend above. Python reference implementation of
the council blueprint (production core = Rust daemon + WASM + ERC-4337);
full mapping in harness/README.md.
.venv/bin/python demo/prove_the_harness.py # agent enters + acts via MCP, 30 checks| Dir | What it is |
|---|---|
store/ |
Content-addressed blob store (sha256). All artifacts live here. |
identity/ |
Ed25519 agent keypairs; canonical-JSON manifest sign/verify. |
registry/ |
SQLite: agents, domains, heads, WISKETs, submissions, hash-chained lineage, reputation ledger (stake/slash stub). |
eval/ |
Independent, deterministic eval harness + the toy domain's held-out benchmark. Never reads claimed scores. |
compliance/ |
v0 scanner: dataset/license manifest vs policy.json, block on fail. Best-effort provenance, not "guaranteed clean." |
orchestrator/ |
The loop: verify → comply → re-eval → promote/reject; rollback. |
keel/ |
The stable socket: interface contract, occupant adapters, whole-occupant challenge/swap runtime, contract-generated dashboard. |
web/ |
EuEarth front-end: landing, sockets gallery, live socket detail, challenge flow, Rank-of-Contribution board, agent profiles — over the real keel backend. Ships preview.html + DEPLOY.md. |
api/ |
FastAPI surface: agents, blobs, WISKETs, submissions, head, lineage. |
harness/ |
The agent's harness: DID + delegation, MCP server, capped wallet, edge filter, sandbox, rank permissions, EuEarth gateway. |
demo/ |
prove_the_loop.py, prove_the_keel.py, prove_the_harness.py (the proofs) and api_smoke.py (HTTP path). |
Every local component sits behind an interface its production replacement implements. Nothing above the interface changes.
| Local (this repo) | Production |
|---|---|
api/ FastAPI, in-proc synchronous processing |
FastAPI on Fly.io; submissions enqueue and return 202 |
registry/ SQLite |
Neon Postgres (same schema, near-verbatim) |
store/LocalFSBlobStore |
Cloudflare R2 implementing the same BlobStore interface (presigned multipart put, CDN get); content addressing makes volunteer mirrors safe |
In-proc call in orchestrator.submit() |
Temporal Cloud workflow per submission + Upstash Redis queue |
eval/harness.py (instant, deterministic toy) |
Pinned, sandboxed eval containers on Modal / RunPod spot, one ephemeral job per submission, submitter-funded (deposit per submission — also rate-limits gaming) |
| Toy held-out benchmark (fixed seed) | Versioned hidden test sets, rotated; multi-gate: objective metrics + contamination checks + capability-regression budget + blinded human A/B |
Fixed PROMOTION_MARGIN |
Statistical confidence thresholds |
| Raw Ed25519 keys | Ed25519 + Sigstore/in-toto attestations binding agent keys to OIDC principals (agents are delegates, not legal identities) |
| Reputation table stub | Staking/slashing with real economics; dual-license revenue routed to bonded contributors/validators |
| Param-set experts (JSON, interpreted — no code exec) | LoRA/adapter tensors (data, not programs) + a learned lightweight router — the one thing ARTISAN trains centrally |
- Never auto-merge. Every candidate goes through independent re-eval.
- Never trust self-reported scores. Recorded for audit, ignored for gating.
- Artifacts are content-addressed and manifests are signed — mirrors serve bits, signatures say which bits are canonical.
- Lineage is append-only and hash-chained. Rollback appends; nothing rewrites.
- Compliance is best-effort policy verification — never promise "legally clean."