Repository navigation
Conversation
Add `Mappings`, a byte stream that stores each token as deltas in about 3 bytes instead of the 24 bytes of a `Token`, and `PackedSourceMap`, a source map that holds its mappings in that form. - `PackedSourceMap::from_json_string` / `to_json_string` convert between VLQ and `Mappings` one token at a time, without building a token array. - `PackedConcatSourceMapBuilder` joins maps by copying bytes. - `SourceMap::into_packed`, `PackedSourceMap::unpack` and `into_sourcemap` convert between the two. Existing types and functions are unchanged.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Merging this PR will regress 2 benchmarks
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b5262f8820
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let name = if token.name_id == INVALID_ID { | ||
| None | ||
| } else { | ||
| let name = delta(token.name_id, end.name_id); | ||
| end.name_id = token.name_id; |
There was a problem hiding this comment.
Ignore unmapped tokens when updating the name baseline
When a token has source_id: None but a name, which both Token::new and SourceMapBuilder::add_token allow, this updates the name baseline even though serialize_packed_mappings emits no name for that unmapped segment. For example, an unmapped token with name ID 1 followed by a mapped token with name ID 0 produces a name delta of -1, so the serialized map fails to parse with an invalid name reference. The existing SourceMap serializer ignores the unmapped token's name and serializes this input correctly. Only retain and update the name when the token is mapped.
Useful? React with 👍 / 👎.
…ceMap So that a consumer can hold a map packed from where it is created to where it is written, without unpacking it to read or change its strings.
Summary
Adds an opt-in compact representation for mappings. Nothing existing changes:
SourceMap,Token,ConcatSourceMapBuilderand the other types keep their API and their code paths.A
Tokenis 24 bytes. A bundler holds most of its maps for much longer than it reads them (the map of a module waits for its chunk, the map of a chunk waits to be written), so the token arrays are a large share of live memory in big builds. This PR adds a form that is about 3 bytes for each token and is decoded only when tokens are needed.New API
Mappings: the tokens of a map as a byte stream.push,from_tokens,tokens()(iterator ofToken),append,last,len,byte_len.PackedSourceMap: a source map that holdsMappingsinstead ofBox<[Token]>.from_json_string/to_json_stringunpack()(borrows the strings) andinto_sourcemap()(moves them) to get aSourceMapfor lookupsnew,from_json,to_data_urlmappings(),tokens(),token_count(),strings(), theget_*andset_*accessors ofSourceMapfor the fields that are not mappings,into_ownedSourceMap::into_packed()andFromimpls in both directions.PackedConcatSourceMapBuilder:with_capacity,from_sourcemaps,add_sourcemap,add_sourcemap_owned,into_sourcemap.How it works
Each token is a tag byte followed by the deltas it needs as zigzag LEB128 varints. The tag says whether the generated line changed, whether the token is mapped, whether it is named, and whether it is on the same source and original line as the previous mapped token. It also carries the generated column delta when that is 0–14. The common token is two bytes.
The deltas follow the rules of the
mappingsfield (column relative within a line, source fields relative to the previous mapped token, name relative to the previous named token). Because of that:PackedSourceMapwrites each VLQ segment straight into the stream, andPackedSourceMap→ JSON writes each token straight back. No token array is built in either direction.Trade-offs
unpack(), which decodes into a normalSourceMap.0.Measurements
Existing benchmarks (
parse,serialize) are unchanged againstmainwithin noise, e.g.parse/real_xlarge83.1 µs → 81.9 µs,serialize/real_large976 ns → 968 ns.New
packedbenchmark,real_xlargefixture:SourceMapPackedSourceMapinto_packed)On a separate set of 73 real library source maps (8.58M tokens), measured with a local harness and not part of this PR, the mappings took 196.3 MiB as tokens and 24.2 MiB packed, and joining them in groups of 64 by byte copy took about 84% less time than joining token arrays.
Tests
tests/packed.rs: round trips on the perf fixtures (same tokens, same JSON, same bytes whether packed from tokens or parsed from JSON), concat equality againstConcatSourceMapBuilder(borrowed, owned and pre-sized), boundary-token dedup, string setters, empty and invalid input.tests/tc39_spec_tests.rs: the packed decoder must accept and reject the same maps asSourceMap::from_json_stringand produce the same JSON.src/mappings.rsfor extreme values andappend.cargo test,cargo clippy --all-targets --all-features,cargo fmtandcargo docare clean.dprint fmtwas not run (not available locally).Notes
Token'ssource_idandname_idfields becomepub(crate)(they were private); no public change.encode_to_stringis split into a thin wrapper over a shared helper so the packed path reuses the JSON writer.