dataset-format.md

Dataset format reference

One JSON object per line (dataset.jsonl), produced by pipeline runs and consumed by validation, deduplication, splitting and training. Decision: ADR-0007.

Record schema

{
  "id": "case-cmd-injection-001",
  "domain": "security",
  "messages": [
    {"role": "user", "content": "Find vulnerabilities\n\n```\n...code...\n```"},
    {"role": "assistant", "content": "{\"verdict\": \"confirmed\", \"cwe\": \"CWE-78\", ...}"}
  ],
  "metadata": { }
}

Field	Type	Rules
id	string	deterministic case-<sha1[:10]> from the case builder
domain	string	domain pack name
messages	list	first message user, last message assistant, non-empty contents
messages[-1].content	string	valid JSON: the canonical structured verdict
metadata	object	provenance, see below

The student sees only messages. Metadata is for tooling and audit.

Multi-turn records (agentic)

The agentic pack emits multi-turn records that interleave assistant tool calls with tool observations, then end on an assistant verdict:

{
  "domain": "agentic",
  "messages": [
    {"role": "user", "content": "task + available tools"},
    {"role": "assistant", "content": "{\"thought\": \"...\", \"action\": \"ls\", \"action_input\": {}}"},
    {"role": "tool", "content": "a.py b.py"},
    {"role": "assistant", "content": "{\"verdict\": \"valid\", ...}"}
  ]
}

Rules beyond the two-turn case: every tool message must immediately follow an assistant message; the last message is still assistant (the verdict). train run --kd rejects multi-turn records (knowledge distillation aligns a single assistant turn), so exclude them from KD datasets.

metadata keys

Key	Type	Description
pipeline	string	pipeline that produced the record
domain	string	domain pack
language	string | null	case language
source	object	{type, repository, commit, path, url, date} from the case
source_date	string | null	ISO date driving contamination tagging
recitation_risk	bool	true when pre-cutoff or unknown date/cutoff
recitation_reason	string | null	pre_cutoff_source, unknown_source_date, unknown_teacher_cutoff
teachers	list	one entry per teacher role: {role, kind, provider, model, family}
judge	object | null	same shape, absent for judge-less runs
teacher_agreement	object	role -> bool: teacher signal vs judge verdict
rag	object	{used, chunks: [{source, doc_id, chunk_id, hash, score}]}
quality	object	{passed, reasons} from the quality gate
created_at	string	ISO timestamp
fallback	bool	present and true on abstention records (opt-in fallback_examples)
fallback_reason	string	gate reasons that triggered the fallback; required when fallback is true
critic_retry	bool	present and true when the record followed a critic escalation
first_pass_reasons	list	gate reasons of the rejected first pass; present with critic_retry

Fallback records (teaching abstention)

With pipeline.fallback_examples: true, a rejected case is written as a two-turn abstention record instead of being dropped:

{
  "id": "case-post",
  "domain": "security",
  "messages": [
    {"role": "user", "content": "<the usual student input>"},
    {"role": "assistant", "content": "{\n  \"verdict\": \"insufficient_information\",\n  \"reasoning\": \"unresolved_disagreement\",\n  \"confidence\": 0.0\n}"}
  ],
  "metadata": {"fallback": true, "fallback_reason": "unresolved_disagreement", "...": "same provenance keys"}
}

The assistant target is a canonical insufficient_information verdict (a value every domain pack accepts) plus the gate reasons, so the student learns to fall back exactly when the ensemble judged the case unreliable. The rejected-case audit trail in data/rejected/ is kept either way. Provider failures never become fallback records. See ADR-0016.

Validation rules (brainforge dataset validate)

• record parses against the schema;
• first message is user, last is assistant, roles in {system, user, assistant, tool}, no empty content;
• each tool message follows an assistant message;
• for two-turn records, assistant content parses as JSON;
• metadata contains source, teachers, quality, recitation_risk;
• metadata.quality.passed is true, unless metadata.fallback is true, in which case metadata.fallback_reason is required instead.

Invalid records are listed with index, id and reasons; the command exits 1.

Case files (input format)

{
  "id": "optional-stable-id",
  "source": {"type": "git", "repository": "repo", "commit": "abc", "path": "tools/x.py", "date": "2026-08-10"},
  "input": {"code": "...", "description": "...", "question": "optional"},
  "metadata": {"language": "python"}
}

Plain source files (.py, .c, .js, ...) are auto-converted: code becomes input.code, language comes from the extension, source.date is the file's last git commit date, and the id is derived from source + content.

Splits

train prepare / dataset split emit:

• train.jsonl, validation.jsonl, test.jsonl: source-grouped 80/10/10 (seeded); one repository never spans two splits;
• test_postcutoff.jsonl: records with recitation_risk: false from all splits; the primary evaluation signal (contamination);
• stats.json: split sizes and the post-cutoff warning.

KD sidecar (<dataset>.kd.jsonl)

Produced by pipeline run --kd (one sidecar record per accepted dataset record) and consumed by train run --kd. One JSON object per line:

{
  "id": "case-cmd-injection-001",
  "model": "glm-5.2",
  "positions": [
    {"token": "{", "logprob": -0.05, "topk": [["{", -0.05], ["\"", -3.2]]},
    {"token": "\n", "logprob": -0.8, "topk": [["\n", -0.8], [" ", -2.1]]}
  ]
}

Field	Type	Description
id	string	dataset record id (matches the main dataset)
model	string	judge model that produced the positions
positions	list	per generated token: {token, logprob, topk}; topk is [[token, logprob], ...]

Notes:

• The tokens of all positions concatenate to the assistant content stored in the dataset record (KD mode stores the teacher's exact output, not a re-serialization, so alignment is exact).
• Top-k is capped by training.kd_topk (default 8).
• Requires a provider that serves logprobs (OpenAI-compatible chat completions with logprobs=true); the mock provider produces word-level positions for dry runs.

See knowledge distillation.