fix(peft_trainer): make scalar-metric host-fetch multi-process safe - #2041
Open
lokic233 wants to merge 1 commit into
Open
fix(peft_trainer): make scalar-metric host-fetch multi-process safe#2041lokic233 wants to merge 1 commit into
lokic233 wants to merge 1 commit into
Conversation
Loss, perplexity and learning_rate are replicated scalars. Under a multi-process / multi-host mesh the underlying jax.Array spans non-process-local (non-addressable) devices, so jax.device_get / np.asarray on it raises "RuntimeError: Fetching value for jax.Array that spans non-addressable (non process local) devices is not possible", crashing the train->metrics logging path on any multi-host / bare-SPMD run. Add a small _addressable_scalar helper (following the existing is_fully_addressable idiom already used in _shard) that, for a *replicated scalar* jax.Array, returns this process's local addressable shard before the host fetch. It only reduces when the local shard shape equals the global shape, so a genuinely sharded non-scalar array is never silently truncated to a partial shard; fully-addressable arrays and plain Python/NumPy scalars pass through unchanged (single-process behavior is identical). Applied at the four scalar host-fetch sites: - MetricsBuffer.loss (buffered-loss mean) - _log_metrics: jax.device_get(loss) and jax.device_get(learning_rate) - _write_metrics._to_np_array (additional scalar metrics: reward, kl, etc.) Validated on a 16-process bare-SPMD RL run on TPU v7x: the metrics flush that previously crashed at jax.device_get(learning_rate) now completes, and training proceeds across many steps.
lokic233
requested review from
abheesht17,
hgao327,
jiangyangmu,
lc5211,
s-noghabi,
sizhit2,
tianshub and
wang2yn84
as code owners
August 30, 2026 04:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the scalar-metric host-fetch in
PeftTrainersafe under a multi-process / multi-host mesh.Why
Loss, perplexity and learning_rate are replicated scalars. Under a multi-process / multi-host mesh the underlying
jax.Arrayspans non-process-local (non-addressable) devices, sojax.device_get/np.asarrayon it raises:This crashes the train→metrics logging path on any multi-host / bare-SPMD run — the training step itself succeeds, but the metrics flush at the end of the step throws.
What changes
Adds a small
_addressable_scalarhelper (following the existingis_fully_addressableidiom already used in_shard) that, for a replicated scalarjax.Array, returns this process's local addressable shard before the host fetch.It only reduces when the local shard shape equals the global shape, so a genuinely sharded, non-scalar array is never silently truncated to a partial shard. Fully-addressable arrays and plain Python/NumPy scalars pass through unchanged — single-process behavior is identical.
Applied at the four scalar host-fetch sites:
MetricsBuffer.loss(buffered-loss mean)_log_metrics:jax.device_get(loss)andjax.device_get(learning_rate)_write_metrics._to_np_array(additional scalar metrics: reward, kl, etc.)Test
Validated on a 16-process bare-SPMD RL run on TPU v7x: the metrics flush that previously crashed at
jax.device_get(learning_rate)now completes, and training proceeds across many steps.peft_trainer.pypy_compile clean and 80-col compliant.