Skip to content

feat: add PackedSourceMap with compact mappings storage - #518

Open
Boshen wants to merge 2 commits into
mainfrom
feat/packed-mappings
Open

Boshen wants to merge 2 commits into
mainfrom
feat/packed-mappings

Conversation

@Boshen

@Boshen Boshen commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds an opt-in compact representation for mappings. Nothing existing changes: SourceMap, Token, ConcatSourceMapBuilder and the other types keep their API and their code paths.

A Token is 24 bytes. A bundler holds most of its maps for much longer than it reads them (the map of a module waits for its chunk, the map of a chunk waits to be written), so the token arrays are a large share of live memory in big builds. This PR adds a form that is about 3 bytes for each token and is decoded only when tokens are needed.

New API

  • Mappings: the tokens of a map as a byte stream. push, from_tokens, tokens() (iterator of Token), append, last, len, byte_len.
  • PackedSourceMap: a source map that holds Mappings instead of Box<[Token]>.
    • from_json_string / to_json_string
    • unpack() (borrows the strings) and into_sourcemap() (moves them) to get a SourceMap for lookups
    • new, from_json, to_data_url
    • mappings(), tokens(), token_count(), strings(), the get_* and set_* accessors of SourceMap for the fields that are not mappings, into_owned
  • SourceMap::into_packed() and From impls in both directions.
  • PackedConcatSourceMapBuilder: with_capacity, from_sourcemaps, add_sourcemap, add_sourcemap_owned, into_sourcemap.

How it works

Each token is a tag byte followed by the deltas it needs as zigzag LEB128 varints. The tag says whether the generated line changed, whether the token is mapped, whether it is named, and whether it is on the same source and original line as the previous mapped token. It also carries the generated column delta when that is 0–14. The common token is two bytes.

The deltas follow the rules of the mappings field (column relative within a line, source fields relative to the previous mapped token, name relative to the previous named token). Because of that:

  • JSON → PackedSourceMap writes each VLQ segment straight into the stream, and PackedSourceMap → JSON writes each token straight back. No token array is built in either direction.
  • Joining two streams is a byte copy. Only the first token, the first mapped token and the first named token of the appended stream are rewritten.

Trade-offs

  • The stream is read from the start only. Lookups need unpack(), which decodes into a normal SourceMap.
  • An unmapped token keeps only its generated position, so its original line and column read back as 0.
  • Parsing and serializing the packed form are slower than the token form (numbers below).

Measurements

Existing benchmarks (parse, serialize) are unchanged against main within noise, e.g. parse/real_xlarge 83.1 µs → 81.9 µs, serialize/real_large 976 ns → 968 ns.

New packed benchmark, real_xlarge fixture:

SourceMap PackedSourceMap
parse JSON 83 µs 104 µs
serialize JSON 38 µs 43 µs
pack (into_packed) – 27 µs
unpack – 26 µs

On a separate set of 73 real library source maps (8.58M tokens), measured with a local harness and not part of this PR, the mappings took 196.3 MiB as tokens and 24.2 MiB packed, and joining them in groups of 64 by byte copy took about 84% less time than joining token arrays.

Tests

  • tests/packed.rs: round trips on the perf fixtures (same tokens, same JSON, same bytes whether packed from tokens or parsed from JSON), concat equality against ConcatSourceMapBuilder (borrowed, owned and pre-sized), boundary-token dedup, string setters, empty and invalid input.
  • tests/tc39_spec_tests.rs: the packed decoder must accept and reject the same maps as SourceMap::from_json_string and produce the same JSON.
  • Unit tests in src/mappings.rs for extreme values and append.

cargo test, cargo clippy --all-targets --all-features, cargo fmt and cargo doc are clean. dprint fmt was not run (not available locally).

Notes

  • Token's source_id and name_id fields become pub(crate) (they were private); no public change.
  • encode_to_string is split into a thin wrapper over a shared helper so the packed path reuses the JSON writer.
  • This PR was written with AI assistance (Claude Code).

Add `Mappings`, a byte stream that stores each token as deltas in about
3 bytes instead of the 24 bytes of a `Token`, and `PackedSourceMap`, a
source map that holds its mappings in that form.

- `PackedSourceMap::from_json_string` / `to_json_string` convert between
  VLQ and `Mappings` one token at a time, without building a token array.
- `PackedConcatSourceMapBuilder` joins maps by copying bytes.
- `SourceMap::into_packed`, `PackedSourceMap::unpack` and
  `into_sourcemap` convert between the two.

Existing types and functions are unchanged.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T07:47:56.979177Z 94dda92 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@codspeed

codspeed Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 2 benchmarks

⚡ 3 improved benchmarks
❌ 2 regressed benchmarks
✅ 15 untouched benchmarks
🆕 17 new benchmarks
⏩ 27 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ lookup_table[real_small] 2.6 µs 2.6 µs -2.1%
❌ lookup_table[real_medium] 2.8 µs 2.9 µs -1.92%
⚡ lookup_token[real_small] 1.3 µs 1.3 µs +4.4%
⚡ lookup_token[real_medium] 1.7 µs 1.7 µs +3.28%
⚡ serialize[real_small] 8.7 µs 8.6 µs +1.47%
🆕 concat N/A 205.6 µs N/A
🆕 pack[real_large] N/A 32 µs N/A
🆕 pack[real_medium] N/A 11.1 µs N/A
🆕 pack[real_small] N/A 10.7 µs N/A
🆕 pack[real_xlarge] N/A 864.3 µs N/A
🆕 parse[real_large] N/A 90.9 µs N/A
🆕 parse[real_medium] N/A 32.8 µs N/A
🆕 parse[real_small] N/A 30.5 µs N/A
🆕 parse[real_xlarge] N/A 2.4 ms N/A
🆕 serialize[real_large] N/A 33.9 µs N/A
🆕 serialize[real_medium] N/A 10.1 µs N/A
🆕 serialize[real_small] N/A 8.4 µs N/A
🆕 serialize[real_xlarge] N/A 902.3 µs N/A
🆕 unpack[real_large] N/A 33 µs N/A
🆕 unpack[real_medium] N/A 7.7 µs N/A
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing feat/packed-mappings (94dda92) with main (fac8357)

Open in CodSpeed

Footnotes

  1. 27 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b5262f8820

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/mappings.rs
Comment on lines +413 to +417
let name = if token.name_id == INVALID_ID {
None
} else {
let name = delta(token.name_id, end.name_id);
end.name_id = token.name_id;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Ignore unmapped tokens when updating the name baseline

When a token has source_id: None but a name, which both Token::new and SourceMapBuilder::add_token allow, this updates the name baseline even though serialize_packed_mappings emits no name for that unmapped segment. For example, an unmapped token with name ID 1 followed by a mapped token with name ID 0 produces a name delta of -1, so the serialized map fails to parse with an invalid name reference. The existing SourceMap serializer ignores the unmapped token's name and serializes this input correctly. Only retain and update the name when the token is mapped.

Useful? React with 👍 / 👎.

…ceMap

So that a consumer can hold a map packed from where it is created to where it is
written, without unpacking it to read or change its strings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant