Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks

This repository contains the code and data to replicate our study "Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks". We test whether standardized commonsense benchmarks predict performance on related downstream tasks, and whether reworked benchmark variants improve that relationship.

The project evaluates 23 language models from six model families on:

• four original/reworked commonsense benchmark pairs;
• eight downstream tasks spanning social/emotional reasoning, pragmatic inference, and event/physical plausibility; and
• three non-commonsense control benchmarks.

It supports benchmark evaluation, log-likelihood confidence analysis, univariate validity analyses, nested regression, leave-one-model-out cross-validation, and leave-one-family-out cross-validation.

Repository layout

analysis/
  confidence/       Log-likelihood margin and confidence analyses
  multivariate/     Ridge, nested-model, LOOCV, and LOFOCV analyses
  univariate/       Correlation, axis-specificity, and rank analyses
data/
  commonsense/      Original and reworked commonsense evaluation sets
  downstream/       Curated downstream evaluation sets
  controls/         Non-commonsense control evaluation sets
evaluation/
  commonsense/      Commonsense benchmark runners
  downstream/       Downstream-task runners
  controls/         Control benchmark runners
results/            Compact final statistical reports

Benchmarks

Group	Tasks
Original commonsense	HellaSwag, Physical IQA, Social IQA, WinoGrande
Reworked commonsense	GoldenSwag, PIQA-RUP, filtered Social IQA, WinoWhat
Social/emotional downstream	CEI, SARC7, False Beliefs
Pragmatic downstream	Implicature, Presupposition, Indirect Requests
Event/physical downstream	TimeDial, TRIP
Controls	BLiMP subsets, GPQA-Diamond, MMLU-Redux subsets

Installation

Python 3.10 or newer is recommended. Create an isolated environment, then install the dependencies:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

GPU evaluation requires a CUDA-compatible PyTorch installation. Some Hugging Face models are gated and require HF_TOKEN to be set in the environment.

Running evaluations

All runners accept a repository root, Hugging Face model identifier, and output directory. For example:

python evaluation/commonsense/hellaswag.py \
  --cs_root "$PWD" \
  --model_name meta-llama/Llama-3.1-8B \
  --output_folder outputs/commonsense/hellaswag/Llama3.1-8B

python evaluation/downstream/trip.py \
  --cs_root "$PWD" \
  --model_name meta-llama/Llama-3.1-8B-Instruct \
  --output_folder outputs/downstream/trip/Llama3.1-8B

The runners write per-item predictions and a scikit-learn classification_report.txt. Commonsense, downstream, and control outputs should follow this layout:

outputs/
  commonsense/<task>/<model>/classification_report.txt
  downstream/<task>/<model>/classification_report.txt
  controls/<task>/<model>/classification_report.txt

Running analyses

Analysis scripts contain a small configuration section near the top for model names and input/output directories. Point the three result roots to the corresponding directories under outputs/, then run, for example:

python analysis/univariate/correlations_cs_downstream.py
python analysis/univariate/correlations_cs_downstream_with_control.py
python analysis/univariate/correlations_cs_downstream_axes.py
python analysis/multivariate/leave_one_family_out.py

The primary analyses use Spearman correlations with bootstrap confidence intervals and Benjamini-Hochberg correction. The multivariate scripts compare control-only models with models augmented by original and/or reworked commonsense composites. LOFOCV holds out all models from one family together while retaining model-level predictions.

Reproducibility notes

• Random seeds are fixed where sampling or bootstrapping is used.
• Model aliases, sizes, families, and Hugging Face identifiers are recorded in the multivariate scripts.
• The compact reports in results/ preserve the final univariate outputs.
• Raw model outputs can be regenerated with the evaluation runners but may differ slightly across hardware and dependency versions.
• Dataset files and code are provided for research reproducibility only.

Citation

If you use this code, please cite the accompanying paper.

TBD