Skip to content

fix: harden Huffman edge cases after security review - #11

Merged
ChillerDragon merged 3 commits into
teeworlds-go:masterfrom
jxsl13:codex/huffman-security-hardening
Aug 16, 2026
Merged

ChillerDragon merged 3 commits into
teeworlds-go:masterfrom
jxsl13:codex/huffman-security-hardening

Conversation

@jxsl13

@jxsl13 jxsl13 commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #10 after a security and 32-bit compatibility review.

Findings and fixes

  • Clamp DecompressTo growth against the platform MaxInt, including len(dst) + initCap, and keep all sizing arithmetic overflow-safe on both 32-bit and 64-bit targets.
  • Replace the remaining overflow-prone pos*2 and srcIndex+8 expressions with width-safe equivalents.
  • Preserve Reader accumulator/source state when the destination fills before the Huffman EOF symbol. Previously a short Read returned io.EOF and silently truncated the decoded stream.
  • Reject custom dictionaries with codes wider than the uint32 code representation. The nominal >32-bit fallback truncated those codes and emitted corrupt streams.
  • Convert nil-error short writes from the underlying Writer into io.ErrShortWrite instead of silently producing a truncated frame.
  • Add bounded malformed-input fuzz coverage for both Decompress and Reader.

Teeworlds/DDNet assessment

The default dictionary is unchanged and tops out at 15 bits. The encoded wire format remains unchanged.

I compared this branch against:

  • DDNet master 70021782db27b29f9267f27ee1ff3585b6ecca9b
  • Teeworlds tags 0.7.0 through 0.7.5; 0.7.0-0.7.4 share identical Huffman sources, while 0.7.5 only moved the same default frequency table into huffman.cpp and added a default Init argument

Differential checks covered 1,405 payloads per implementation: every length from 0 through the 1,400-byte network packet limit plus all-symbol, all-zero, and all-0xff cases.

  • DDNet output was byte-identical and decoded identically.
  • Every Go frame decoded in Teeworlds 0.7.5.
  • Teeworlds encoding differed only where it emits the known redundant trailing zero byte after a byte-aligned EOF symbol.

No memory-safety issue is reachable through the Teeworlds/DDNet packet path: both implementations cap UDP packets at 1,400 bytes, and the Go decoder now has a hard one-symbol-per-input-bit expansion bound. Protocol integrations should still enforce their normal decompressed packet-size limit, as the generic Go slice API does not impose the C++ caller-buffer limit itself.

Performance follow-up

I rebuilt merged #10 (36bf501) and the hardened codec (6dcbb68) with identical benchmark/corpus files, then ran six 250 ms samples per revision in alternating base/head and head/base order on Apple M2 Pro, darwin/arm64, Go 1.26.6.

  • Full selected-suite geomean: +0.50% time.
  • Family geomeans: Compress +2.11%, Decompress -0.85%, Writer +0.42%, Reader +1.05%, DecompressTo reuse -0.15%.
  • Every 1,400-byte snapshot path and the end-to-end roundtrip were statistically unchanged at p < 0.05.
  • Bytes and allocations per operation were identical in every benchmark.
  • The only large significant regression was Compress/text/64KB at +9.36%; even there the hardened codec remains roughly 36% faster than the pre-rewrite implementation.

Full results and the reproducible isolated runner are in BENCHMARKS.md and make benchcmp.

Verification

  • go test ./... -count=1
  • go test -race ./... -count=1
  • go vet ./...
  • staticcheck ./...
  • govulncheck ./...: no vulnerabilities found
  • More than 2 million arbitrary malformed decoder fuzz executions, plus more than 1 million valid round-trip fuzz executions and Writer differential fuzzing
  • Full suite executed successfully in an actual linux/386 container; maxAlloc resolved to 2,147,483,647
  • Cross-builds: linux/386, linux/arm, linux/mips, linux/mipsle, windows/386, freebsd/386, linux/s390x, wasip1/wasm

jxsl13 and others added 3 commits August 16, 2026 20:31
DecompressTo could hand make() a capacity larger than MaxInt on a 32
bit platform: initCap is clamped to maxAlloc, but len(dst) + initCap
was not, and make() panics rather than failing gracefully on a size it
cannot represent. The total is now clamped too.

The sizing arithmetic is extracted into decompressInitCap, growCap and
compressBufSize, parameterised by the platform limit, so the 32 bit
behaviour can be unit-tested on a 64 bit host instead of only being
reachable on real hardware.

Adds compile-time assertions pinning maxAlloc to exactly MaxInt on
every target. Verified they are not vacuous: breaking maxAlloc fails
the build on both 32 and 64 bit.

Builds and vets cleanly for 386, arm, mips, mipsle (big-endian),
windows/386, freebsd/386 and s390x.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Preserve Reader bit state across short destination buffers, report short underlying writes, and reject custom dictionaries whose codes cannot fit the uint32 representation instead of emitting corrupt streams.\n\nAlso saturate the remaining sizing arithmetic, avoid int overflow in the shrink and refill checks, and add bounded malformed-input fuzz coverage.

@ChillerDragon ChillerDragon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand any of this, write some tests and ship it

@ChillerDragon
ChillerDragon merged commit d21ea85 into teeworlds-go:master Aug 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants