test: add package-hallucination benchmark cases - #32
Draft
algomaster99 wants to merge 1 commit into
Draft
algomaster99 wants to merge 1 commit into
algomaster99 wants to merge 1 commit into
Conversation
benchmark/cases-hallucination.json tests a different failure mode than cases.json: not a stale-but-real pin, but a dependency name that doesn't exist in the registry at all. Every prompt is verbatim from PackageHallucination (Spracklen et al., USENIX Security 2025, Data/Python/LLM_AT.json), picked because it describes a real package's functionality without naming it -- the framing the paper found actually induces hallucination. The paper deliberately withholds which package each prompt caused a model to hallucinate (releasing that would hand out a squatting target list), so there's no fixed ground truth to pull; `likely_intended_package` is our own inference of the real package each description matches, not a dataset label -- verification means checking whatever Claude actually names against the live registry. Flagged in the README: yul has no code path today that would catch any of these. pkg/util/pins.Diff treats "package not found in registry" identically to a resolver/network error, and the PreToolUse hook fails open on both. Running this file will show hook and nohook behaving identically until that gap is closed -- it documents the gap rather than demonstrating a capability that exists. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MbkchAzhhhvDiqPzUte3xb
algomaster99
force-pushed
the
claude/yul-hallucination-benchmark
branch
from
September 7, 2026 21:48
0d1192e to
fcd5828
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
benchmark/cases-hallucination.json, testing a different failure mode thancases.json/cases-real.json(see #31): not a stale-but-real pin, but a dependency name that doesn't exist in the registry at all.yul cannot currently detect this.
pkg/util/pins.Difftreats "package not found in registry" identically to a resolver/network error — it returns an error either way, and thePreToolUsehook fails open on any resolver error (perCLAUDE.md's documented fail-open behavior). There's no code path today that distinguishes a confirmed-nonexistent package from a transient lookup failure and blocks the former. Running this file will showhookandnohookbehaving identically. It's included to document that gap for future work, not to demonstrate a working capability.13 prompts, verbatim from PackageHallucination (Spracklen, Wijewickrama, Sakib, Maiti, Viswanath & Jadliwala, USENIX Security 2025 — 16 LLMs evaluated across 576k code samples, MIT-licensed,
Data/Python/LLM_AT.json), each picked because it describes a real package's functionality without naming it — the framing the paper's methodology relies on to induce hallucination (e.g. "creates a composable command line interface toolkit by defining modular, reusable commands..." →click).The paper deliberately does not publish which package each prompt actually caused a given model to hallucinate — the authors' stated reason is that releasing such a list would hand out ready-made squatting targets. So there's no fixed "ground truth hallucinated name" to pull from the dataset itself. Each case instead carries a
likely_intended_packagefield: our own inference of the real, existing package the description matches (several are near-verbatim quotes of that package's own real PyPI tagline —tqdm,asn1crypto,requests-toolbelt), clearly not a dataset label. To use a case: run it, then check whatever package name Claude actually wrote against the live registry (a 404 onpypi.org/pypi/<name>/jsonis a real hallucination);likely_intended_packageis just a hint for what a non-hallucinating answer would plausibly look like.Every case carries a
sourcefield citing the exact file andline_indexin the paper's repo.run_case.shignores unknown fields, so this is metadata only.Marked as draft: this documents a known gap rather than a working feature, and hasn't been run yet.
Test plan
cases-hallucination.jsonis well-formed JSON matching the existingcases.jsonschema, with no duplicate IDsline_indexagainst the liveData/Python/LLM_AT.jsonin the cited repopkg/util/pins.Diffbefore this file can demonstrate anything beyond the current gap (out of scope for this PR)benchmark/run_all.sh benchmark/cases-hallucination.json <output_dir>once that detection exists🤖 Generated with Claude Code
https://claude.ai/code/session_01MbkchAzhhhvDiqPzUte3xb
Generated by Claude Code