You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
An Apple M2 Pro Metal Q4_K sweep compared the production 2-row SIMD-group kernel with exact 4-row and 8-row variants. Both candidates were bit-identical to control, including odd tails, and mutated zero input bytes.
A conventional grouped Go benchmark (-benchtime=300x -count=4, all control samples followed by all candidate samples) made rows=8 appear roughly 1.19x faster at K2048/N2048 and about 1.05x faster at K5632/N2048. That apparent gain did not survive an order-alternated fresh-process protocol using the same binary, 7 pairs per campaign, 3 campaigns, and the second sample from each count=2 invocation.
Rows=8 results:
K2048/N2048: 1.015x, 1.015x, 0.976x
K5632/N2048: 0.963x, 0.984x, 0.982x
The dominant K2048/N5632 shape was already neutral-to-negative in the grouped screen. No candidate shipped.
Generalizable implication
GPU tile and occupancy variants can move thermal state and clock residency enough that benchmark subtest grouping creates a false shape-specific win, even with hundreds of iterations and multiple count samples. A detector or benchmark reviewer should flag performance claims where variant samples are grouped by sub-benchmark order. Require arm alternation at the process or invocation boundary, discard each arm first sample, and report every campaign rather than pooling the samples. This extends FIRST-BENCHMARK-SAMPLE-IS-NOT-COMPARABLE-001 to cross-variant GPU ordering.
Activity