Benchmarks

Benchmarks

Karta is evaluated as a long-horizon agent-memory system. Scores are research signals, not stable release claims.

Current headline scores

Benchmark	Status	Current result
BEAM 100K	active tracking	61.6% P1, 2026-04-14
LOCOMO	in progress	query-phase hang under debug; no validated score
LongMemEval-S	in progress	oracle temporal partial: 50 questions, 50%

BEAM 100K

BEAM 100K is the main development target.

• 20 conversations.
• 400 questions.
• 10 memory abilities.
• Scored by BEAM nugget LLM judge.
• Single-run arithmetic mean of per-question rubric scores.

Current P1 score: 61.6%, up from 53.0% P0.

Per-ability P1 snapshot:

Ability	Score
Preference Following	92%
Contradiction Resolution	74%
Temporal Reasoning	71%
Instruction Following	69%
Summarization	66%
Multi-session Reasoning	65%
Abstention	62%
Information Extraction	59%
Knowledge Update	43%
Event Ordering	35%
Overall	61.6%

Reproduce BEAM 100K

Requires real LLM credentials.

BEAM_DATASET_PATH=data/beam-100k.json   cargo test --release -p karta-core --test beam_100k     beam_100k_full -- --ignored --nocapture

Fast deterministic regression tests

No API keys needed:

cargo test --workspace --lib
cargo test -p karta-core --test synthetic_memory_eval
cargo test -p karta-core --test activate_retrieval_invariants

Reproducibility rules

• Published numbers are single runs, not seed averages.
• Treat deltas under ~3pp as likely noise unless confirmed.
• Judge model can swing absolute numbers.
• Full BEAM 100K runs take 10–12 hours on good hardware.
• Use .results/ for local run logs; it is gitignored.
• Do not report benchmark numbers from incomplete or panicked runs.

Source docs

• benchmarks/README.md
• benchmarks/beam-100k.md
• docs/retrieval-plan.md