Replace candle with llama.cpp for property extraction

[disconsented]

Note

Written by a stochastic parrot. This issue was drafted end to end by Claude
Opus 5 on 2026-09-13.

What to trust: every number below was measured on this hardware and the raw JSON
for each run is committed in bench-out*/. Nothing here is recalled from training.

What to check: the synthesis, the judgement calls, and any sentence that sounds
confident. The model changed its own conclusions at least four times during the
session when tested properly, including twice about things it had already told the
user were settled.

TL;DR

candle 0.11 can only correctly run one of the three models tested. Switching to
llama.cpp with Gemma 4 E4B at q4_0 gives a 74% cut in peak memory and a 9×
increase in generation throughput, and makes malformed JSON structurally impossible.

Branch: feat/llamacpp-extraction-bench (4 commits on top of feat/updated_ui).
Runbook: classification/notes/llamacpp-runbook.html.

---

Numbers

Five RimWorld workshop items, two extraction prompts, greedy sampling.
Ryzen 9 7950X3D at 16 threads. Target hardware is a Xeon Gold 6210U.

	candle · Mistral 7B · F16	llama.cpp · Gemma 4 E4B · q4_0	change
Weights on disk	13.5 GiB	4.80 GiB	-64%
Peak RSS	27.3 GiB	7.13 GiB	-74%
RSS while idle	27.3 GiB	0.18 GiB	-99%
Prompt processing	35 tok/s	273 tok/s	7.8×
Generation	2.07 tok/s	19.3 tok/s	9.3×
One item, both prompts	~220 s	~27 s	8×
Valid JSON	6 / 6	24 / 24

Two things drive the memory figure. The published checkpoints are BF16 and candle has
no BF16 matmul on CPU (cpu_backend/mod.rs:1383 gates to F16/F32/F64), so it converts
at load and holds both the mmap and the converted copy. llama.cpp mmaps the file as it
is. Then --sleep-idle-seconds releases the model entirely after idle and faults it
back in on demand, for a measured 1.34 s wake penalty.

Every model tested, including the ones that did not work

model	runtime	disk	peak RSS	gen tok/s	valid JSON
Mistral 7B Instruct	candle f16	13.5 GiB	27.3 GiB	2.07	6/6
Qwen3-VL 8B	candle f16	15.2 GiB	53.1 GiB	2.03	4/6
Qwen3-VL 8B + image	candle f16	15.2 GiB	53.6 GiB	1.95	3/6
Gemma 4 E4B	candle f16	13.9 GiB	17.7 GiB	4.01	0/6
Gemma 4 E4B	llama.cpp bf16	14.0 GiB	9.42 GiB	6.70	6/6
Gemma 4 E4B	llama.cpp q4_0	4.80 GiB	7.13 GiB	19.3	6/6

Gemma 4 under candle looks fastest and smallest. It produced an empty string every
time. Qwen3-VL's 53 GiB is almost entirely an empty KV cache (see defects).

---

Decisions

decision	why
llama.cpp over candle	Not a preference. candle 0.11 cannot run 2 of the 3 models at all.
gemma-4-E4B-it-qat-q4_0	Quantization-aware-trained by Google, so 4-bit holds quality a community post-training quant of the same width would not. Do not swap for a generic Q4_K_M expecting an upgrade.
Single pass, text only	Attaching the preview costs ~280 prompt tokens and made extractions shorter and more generic on every item.
response_format: json_schema	Compiles to a GBNF grammar and constrains sampling, so a markdown fence is not a reachable token. Removes the malformed-JSON class outright.

Rejected: telling the model to output bare JSON

It works, and it costs content. The model reads "respond with the JSON object only"
as an instruction to be brief.

variant	fenced	features	output tokens
baseline	yes	16	148
respond with the JSON object only, no prose	no	2	22
do not wrap the JSON in code fences	no	2	22
response_format: json_schema	no	16	148

Any trailing instruction about output format triggers it, so the milder wording is no
safer. The grammar leaves the prompt untouched and matches baseline exactly.

Deferred: transcribe-then-extract for images

The vision path works. Asked to transcribe a preview, the model read version tags, CJK
glyphs and 8-pixel colonist name labels, while a no-image control answered NO IMAGE.
every time.

Attaching images to the extraction prompt does nothing at all. Transcribing them in a
separate call and appending the text does a great deal:

item	text only	images attached	transcribe then extract
Rustic Workbenches	3 features	2	15
Gravship Ch.2	12 features	14	34 (22 absent from the prose)

The cost is 2.4× wall time on image-heavy mods, and it needs a routing rule to decide
which mods qualify. Worth revisiting, not worth blocking on. Note that Vanilla Expanded
mods put their entire content listing in infographics, and they are a large slice of
the corpus.

---

Also on the branch

Three changes unrelated to the model choice, kept as separate commits.

• fix(build) .cargo/config.toml forced avxifma, avxneconvert, avxvnni,
  avxvnniint8 and avxvnniint16. Zen 4 implements none of them, so anything that
  used one died with SIGILL. serde_json's float formatter did. Repo-wide, nothing to
  do with ML.
• perf(classification) MKL off by default. On Zen 4 it decoded at half the rate
  of candle's own gemm backend with no prefill advantage to trade for it. The root
  crate enabled it explicitly via ml-classification, so flipping the feature default
  alone changed nothing.
• fix(prompts) features.txt gained a sibling-noun rollup rule. Every existing
  example collapsed modifier and noun ("steel sword" to "swords"); none collapsed
  sibling nouns into a parent, so one mod listed seven weapon kinds separately.
  23 features became 12.

---

Follow-up

• Replace PipelineRunner / TextGeneration with an HTTP client against
  /v1/chat/completions. ExtractionActor, MLProperties and the prompts are
  unchanged; the actor's serialised mailbox now protects the single server slot.
• Drop candle-*, intel-mkl-src, tokenizers and hf-hub from
  classification/Cargo.toml.
• Delete sanitise_output (classification/src/lib.rs:245). Note it panics today
  on any answer without a closing brace: find('}').unwrap_or(len) with an
  inclusive range slices one byte past the end. Empty and cap-truncated answers
  both hit it, and run_pipeline calls it unguarded.
• Benchmark on the Cascade Lake target. The q4_0 figures should transfer because
  the VNNI integer path is identical on both chips. Absolute throughput will not:
  the 6210U has six memory channels against this desktop's two.
• Settle model distribution. No size constraint forces this either way: 4.8 GB is
  well under what registries allow. Baking it into the image keeps deploys a single
  artifact and costs one cached layer; a mounted volume keeps CI pushes small and
  lets the model change without a rebuild. Verify
  sha256:676c35070db6dbe52f93e9c864ee0fba4eddea94b9c875d9cb10daff453fbaee
  whichever way it arrives.

---

Upstream defects found in candle 0.11 (probably belongs in its own issue)

• qwen3_vl/text.rs:285 builds lm_head with linear, requiring a bias tensor that
  no Qwen3 checkpoint carries. Should be linear_no_bias. Nothing loads without a
  shim.
• qwen3_vl/text.rs:144 sizes its KV cache to max_position_embeddings and grows by
  that whole amount in one step, so the first token allocates all 262,144 positions.
  That is 38.7 GB regardless of prompt length. qwen3.rs uses a concatenating cache
  and has no such cost.
• gemma4 cannot load any published Gemma 4 checkpoint. The text model never reads
  the per-layer embeddings or the per-layer layer_scalar (measured 0.061 to 0.887,
  mean 0.60, so ignoring them is not benign), and the vision tower asks for
  q_proj.weight where the checkpoint stores the clipped form. The upstream example
  at candle-examples/examples/gemma4/main.rs:247 defaults to google/gemma-4-E4B-it
  and fails at load on both its text and multimodal paths.