vlm-exam

Benchmark suite for Vision Language Models. Compare accuracy, cost, and speed across frontier VLMs on standardized visual tasks.

Supported tasks

• VQA / OCR -- visual question answering and optical character recognition
• Object Detection -- bounding box prediction evaluated with COCO-style mAP

Leaderboard

The results/ directory holds the raw benchmark outputs (one JSONL file per run) and is the single source of truth for the numbers below. Regenerate the charts at any time with vlm-exam leaderboard.

Counting

Extraction

Identification

Reasoning

Object Detection

Stricter IoU thresholds: mAP@75 | mAP@50:95

Supported providers

• Anthropic (Claude)
• Google (Gemini)
• OpenAI (GPT)
• OpenRouter (any OpenAI-compatible vision model, e.g. Qwen 3.7 Plus, GLM 5V Turbo)

Installation

pip install vlm-exam

Or install from source:

git clone https://github.com/roboflow/vlm-exam.git
cd vlm-exam
pip install -e ".[dev]"

Quick start

Set your API keys (or place them in a .env file):

export ANTHROPIC_API_KEY=...
export GOOGLE_API_KEY=...
export OPENAI_API_KEY=...
export OPENROUTER_API_KEY=...

Run a VQA benchmark

Expects a dataset directory containing an annotations.jsonl file with image, prefix (question), and suffix (answer) fields.

vlm-exam run \
    --task vqa \
    --models claude-fable-5,gemini-3.5-flash,gpt-5.5 \
    --effort high \
    --dataset-directory data/vqa/train

Use an LLM judge as a fallback when strict answer matching fails:

vlm-exam run \
    --task vqa \
    --models gpt-5.5 \
    --effort low \
    --dataset-directory data/vqa/train \
    --match-mode judge \
    --judge-model gemini-3.5-flash

Run a detection benchmark

Expects a COCO-format dataset directory containing an _annotations.coco.json file alongside the images.

vlm-exam run \
    --task detection \
    --models gemini-3.5-flash,gpt-5.5,claude-fable-5 \
    --effort low \
    --dataset-directory data/detection/train

Useful options:

• --max-samples 10 limits the number of processed images (handy for smoke tests).
• --prompt-classes image (default) lists only the classes present in each image's ground truth; --prompt-classes all lists every dataset class.

Summarize results

Accuracy, token usage, and cost tables across all saved runs:

vlm-exam report --results-directory results

Dataset-level mAP@50, mAP@75, and mAP@50:95 for detection runs:

vlm-exam detection-report \
    --results-directory results \
    --dataset-directory data/detection/train

Visualize detection predictions

Side-by-side ground truth vs. prediction cards with per-image mAP@50:

vlm-exam detection-visualize \
    --results-file results/detection_gemini-3.5-flash_low_20260707_122136.jsonl \
    --dataset-directory data/detection/train \
    --output-directory visualizations/detection \
    --max-images 20

--label-mode controls box labeling: labels draws class names on the boxes, legend draws boxes only with a color legend below the images, and auto (default) picks based on label density.

Generate leaderboards

Regenerates leaderboard charts for all locally saved runs (VQA accuracy plus detection mAP@50 / mAP@75 / mAP@50:95 per effort level):

vlm-exam leaderboard \
    --results-directory results \
    --dataset-directory data/detection/train \
    --output-directory visualizations/leaderboards

Python

from vlm_exam import load_config, create_provider, create_task, run_benchmark

config = load_config()
task = create_task("vqa")
samples = task.load_samples("/path/to/vqa/dataset")
provider = create_provider("anthropic", model="claude-fable-5")

results = run_benchmark(task=task, provider=provider, samples=samples, effort="high")

Configuration

Model definitions, pricing, lab branding, detection coordinate formats, and optional fallback routes live in src/vlm_exam/configs/models.yaml. Add a new model by editing this file -- no code changes required for single-route models.

Each model may declare detection_coordinate_format (normalized_1000, pixel, or normalized_1000_xyxy) for its native grounding convention. When omitted, the primary route's provider supplies a default.

For rate-limit resilience, list multiple routes in priority order. FallbackProvider fails over on 429/quota errors and sticks to the next route for the rest of the run. Example:

  gemini-3.1-pro-preview:
    detection_coordinate_format: normalized_1000
    routes:
      - provider: google
      - provider: openrouter
        provider_model_id: google/gemini-3.1-pro-preview

For OpenRouter-only models, set provider: openrouter (legacy syntax) or a single routes entry, plus provider_model_id with the OpenRouter slug (e.g. qwen/qwen3-vl-235b-a22b-instruct). The short YAML key appears in result filenames and leaderboards.

License

Apache 2.0. See LICENSE.