JIMMY

Turning a cheap, dumb, unlimited chatbot into a reliable tool — by measurement, not by faith.

Jimmy is a free, unlimited chatbot backed by llama3.1-8B, served from dedicated hardware (chatjimmy.ai) at ~13k tokens/s. This project is a systematic study of what such a model is actually good for — run as a lab, where every idea earns a number against a pre-declared threshold, and dead ideas are kept with the number that killed them.

🇬🇧 English (canonical) · 🇵🇱 Polska wersja na branchu PL

---

The thesis

Jimmy is dumb per-sample, but nearly free per-sample. So stupidity is compensable by volume everywhere verification is cheaper than generation.

The lever is concurrency (~38 req/s), not model speed — the network dominates (RTT ~134 ms), so "13k t/s" is not the budget; requests per second is, and it scales with parallelism.

The laws (confirmed across experiments)

These are the sieve every new idea is run through — see FINDINGS.md for the full derivation.

1. Volume fixes variance, not bias. Best-of-N / voting turns an unreliable 8B into a reliable tool iff its errors are random and a cheap verifier or aggregator exists. Systematic errors (Jimmy consistently misreads a task) are unfixable by more samples.
2. Volume helps optimization, not exploration. Sampling is spiked on modes — great for picking the best of N, poor for covering a categorical space. Exploration is unlocked only by an external accumulating state + novelty pressure (coverage 18% → 80%).
3. A loop is worth it only for accumulation, not optimization. When the optimum fits in a single output, best-of-N wins and the loop adds nothing. When the target is larger than one generation, an accumulating loop is the only thing that works — see HOLY_GRAIL.md.

Hard limits of Jimmy (measured — CHARACTERIZATION.md)

Property	Value	Design consequence
Determinism @ topK=1	identical bytes	memoizable, reproducible, cacheable
Diversity @ topK=8	distinct-rate 1.0	best-of-N is alive
topK ceiling	topK ≥ 16 → HTTP 500	hard cap; diversify via topK=8 + prompt perturbation
max_tokens/stop/temperature/seed	silently ignored	control length/stop via prompt, not API
Context ceiling	~6k prefill OK, ≥8k → empty reply	chunk map-reduce ≤ ~1.2k tokens
Concurrency knee	conc=8 → 37.8 req/s, 0 errors, p50=134 ms	budget ≈ 38 req/s; lever = concurrency

Quick start

pip install httpx                                    # the only runtime dependency

python3 -m jimmy.client                              # smoke-test (1 request)
python3 experiments/001-majority-vote-math/run.py    # reproduce an experiment

# practical tool: summarize a document of any length
tools/summarize.py README.md --style tldr
cat long.txt | tools/summarize.py --map-only > notes.md

Repository layout

jimmy/client.py     # the ONLY HTTP chokepoint: async pool, concurrency cap, retry, <|stats|> parsing
jimmy/eval.py       # cheap deterministic verifiers: JSON, exact-match, voting, distinct-rate, usability
experiments/NNN-*/  # hypothesis.md · run.py · results.* · verdict.md
tools/              # validated patterns packaged as shell commands (stdin→stdout)
CHARACTERIZATION.md # measured properties of Jimmy (Phase 0)
FINDINGS.md         # distilled laws — the sieve for every new idea
HOLY_GRAIL.md       # the "emergent loop" goal, resolved
LOG.md              # append-only propose/evaluate/dispose journal

The client exposes ask, map_prompts (N different prompts in parallel) and sample_n (N samples of one prompt — the core of best-of-N / voting), each returning parsed <|stats|> telemetry for free. All API traffic goes through this one module; the safe-use policy (MAX_CONCURRENCY=8, RPM_CEILING=2000, retry×3) is enforced in code, not in intentions.

Experiments

Method: propose → evaluate → dispose → repeat → note. Each experiment freezes a metric and a threshold before running, then produces a verdict. Negative results are half the product.

Legend: ✅ KEEP · ⚠️ PARTIAL / conditional · ❌ DISPOSE · 🔀 SPLIT · 💡 counter-intuitive finding · 🧭 capability boundary · 🔬 measurement

#	Experiment	Result
000	Characterization (Phase 0)	🔬 Measured Jimmy's properties → CHARACTERIZATION.md
001	Majority-vote on math	✅ KEEP — self-consistency 64% → 92% · verdict
002	Structured JSON extraction	✅ KEEP (sharp boundary) — validity 100%, semantics don't · verdict
003	Candidate generation	🔀 SPLIT — selection KEEP, coverage DISPOSE; anti-mode accumulation KEEP · verdict
004	Map-reduce over a long doc	✅ KEEP — beats context limit, recall 0 → 100 · verdict
006	Holy Grail as a GA	❌ DISPOSE (with diagnosis) — loop adds nothing when optimum ⊂ one output · verdict
007	Open-ended accumulation	❌ DISPOSE — accumulation saturates in open space too · verdict
008	Asymmetric critic	❌ DISPOSE — critic improves draft but loses to equal-budget best-of-N · verdict
009	Jimmy as an automaton	❌ DISPOSE spatial hypothesis, but an emergent discovery · verdict
010	FS-tree map-reduce	💡 Counter-intuitive — for holistic classification, don't decompose · verdict
011	Bash code generation	❌ DISPOSE decomposition — best-of-N wins, Jimmy-reduce breaks · verdict
012	Big input vs targeted context	💡 Relevance & brevity beat size (+55 vs +11) · verdict
013	Hierarchical summarization	⚠️ PARTIAL KEEP — pipeline works, reduce must be chunked too · verdict
014	Web / HTML ops	🧭 Capability boundary — mechanics → regex, semantics → Jimmy · verdict
015	Meta-prompting	❌ DISPOSE — names the right ideas, can't package a working scaffold · verdict
016	Recursive decomposition	⚠️ PARTIAL KEEP — defensive technique wins (+35), deep recursion doesn't · verdict
017	Steps → shell commands	⚠️ KEEP for form, not intent — 96% run, ~20% do the wrong thing · verdict
018	Command analysis	💡 describe→compare repairs judgment (+30); direct judge rubber-stamps · verdict
019	Logic benchmark	🧭 Propositional OK (0.88), quantifiers/higher-order ≈ chance · verdict
020	ArchWiki → manpage	✅ KEEP — hybrid Jimmy content + deterministic troff · verdict
021	Manpage → tldr	⚠️ PARTIAL KEEP — works for tar, unstable for journalctl · verdict
022	Synthetic training data	✅ KEEP (sharp scope) · verdict

The Holy Grail

The project's stated goal was "an infinite Jimmy-based loop with emergent properties." It has a concrete, empirical answer — and not the expected one: the Grail is accumulation, not optimization. An emergent loop is worth building iff it grows a structure larger than any single generation, held in external state (coverage 18% → 80%). Full resolution, the validated recipe, and the named open frontier: HOLY_GRAIL.md.

Tools

Validated patterns, packaged as Unix-style commands (input from file/stdin, result on stdout, diagnostics on stderr). See tools/README.md.

• summarize.py — summarize a document of any length (map-reduce + hierarchical reduce, silent-truncation detection, ~20 flags). Built on experiments 004 + 013.

Safe-use policy — "we love Jimmy, we don't abuse him"

Enforced in jimmy/client.py, not in intentions: MAX_CONCURRENCY=8 (the measured zero-error point), RPM_CEILING=2000, retry×3 with backoff. Tools clamp any user override back to these limits.

For contributors / AI agents

CLAUDE.md documents the architecture, conventions, and hard constraints for anyone (human or agent) working in this repo.

---

_{Research project. All figures are reproducible from the run.py in each experiment folder. Language: main is English (canonical); a Polish snapshot is preserved on the PL branch.}