fix(gemma3): apply rope_scaling to global attention layers - #1709
Conversation
Gemma 3 ships rope_scaling (linear, factor 8) in text_config and applies it to the global-attention layers only. The Attention module declared rope_scaling on TextConfig but built a plain nn.RoPE without it, so global layers ran unscaled positions. The error is negligible at short context and compounds with length: on a 2916-token prompt gemma-3-4b-it-4bit picks a different greedy token than the reference implementation and degenerates into a repetition loop. Use the existing rope_utils.initialize_rope, matching gemma4 in this repo and gemma3_text in mlx-lm. Sliding layers keep unscaled local RoPE. gemma3_text inherits the fix through gemma3.language.LanguageModel. gemma3n carried the same shape and is fixed alongside; with no rope_scaling declared the behaviour is unchanged (scale 1.0).
|
Follow-up: I swept the rest of the repo for the same shape and found ten more models that declare The two are the same defect class and can be reviewed together or merged in either order; they touch disjoint files. This one is separate because it is verified end to end against a real gemma-3 checkpoint (patched mlx-vlm reproduces mlx-lm's reference logits exactly and a served repetition loop disappears), whereas #1710's evidence for |
Problem
Gemma 3 ships
rope_scalingintext_configand applies it to the global-attention layers only:TextConfigdeclares the field, butAttention.__init__builds a plainnn.RoPEand never passes it, so global layers run unscaled positions (effectivescale=1.0where the model expects1/8).The error is negligible at short context and compounds with length. On
mlx-community/gemma-3-4b-it-4bit, first-token logits for a 2916-token prompt (mlx-vlm vs the referencemlx_lm.models.gemma3):Từ(correct continuation)ĐốiGreedy decoding therefore picks a different token and degenerates into a repetition loop, while the same weights under
mlx_lm.generateanswer correctly. Short prompts are unaffected (top-100 logit rank-correlation 0.96 at 195 tokens vs 0.53 at 2916), which is why this hides in ordinary use.Fix
Use the existing
rope_utils.initialize_rope, matchinggemma4in this repo andgemma3_textin mlx-lm. Sliding layers keep unscaled local RoPE; scaling applies to global layers only.After the fix, mlx-vlm reproduces the reference logits exactly:
Scope
gemma3— the reported defect;gemma3_textinherits the fix throughgemma3.language.LanguageModel.gemma3n— same shape (declaresrope_scaling, built a plainnn.RoPE). With norope_scalingin the config the behaviour is unchanged (scale=1.0), so this is a no-op for current checkpoints and correct if one declares scaling.Tests
Two regression tests added to
test_models.py:test_gemma3_rope_scaling_applies_to_global_layers— asserts global layers get1/factor, sliding layers stay1.0. Fails onmainwithAssertionError: 1.0 != 0.125.test_gemma3_without_rope_scaling_is_unscaled— asserts the no-scaling path is untouched.python -m unittest mlx_vlm.tests.test_modelspasses (all 20 gemma tests included).