Skip to content

Interleave GPU tile variants: grouped count runs fabricated a Q4_K rows-per-SIMD win #865

Open

Description

@jxsl13

Finding

An Apple M2 Pro Metal Q4_K sweep compared the production 2-row SIMD-group kernel with exact 4-row and 8-row variants. Both candidates were bit-identical to control, including odd tails, and mutated zero input bytes.

A conventional grouped Go benchmark (-benchtime=300x -count=4, all control samples followed by all candidate samples) made rows=8 appear roughly 1.19x faster at K2048/N2048 and about 1.05x faster at K5632/N2048. That apparent gain did not survive an order-alternated fresh-process protocol using the same binary, 7 pairs per campaign, 3 campaigns, and the second sample from each count=2 invocation.

Rows=8 results:

  • K2048/N2048: 1.015x, 1.015x, 0.976x
  • K5632/N2048: 0.963x, 0.984x, 0.982x

The dominant K2048/N5632 shape was already neutral-to-negative in the grouped screen. No candidate shipped.

Generalizable implication

GPU tile and occupancy variants can move thermal state and clock residency enough that benchmark subtest grouping creates a false shape-specific win, even with hundreds of iterations and multiple count samples. A detector or benchmark reviewer should flag performance claims where variant samples are grouped by sub-benchmark order. Require arm alternation at the process or invocation boundary, discard each arm first sample, and report every campaign rather than pooling the samples. This extends FIRST-BENCHMARK-SAMPLE-IS-NOT-COMPARABLE-001 to cross-variant GPU ordering.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions