Skip to content

Fix unsigned MEDIUMINT binlog values >= 2^23 - #463

Open
driv3r wants to merge 2 commits into
mainfrom
fix-unsigned-mediumint-binlog
Open

driv3r wants to merge 2 commits into
mainfrom
fix-unsigned-mediumint-binlog

Conversation

@driv3r

@driv3r driv3r commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Fixes #299. Supersedes #301.

Problem

go-mysql decodes MYSQL_TYPE_INT24 as a sign-extended int32 (mysql.ParseBinaryInt24). Our unsigned fix-up in NewBinlogDMLEvents converts it with uint32(v), so an unsigned MEDIUMINT of 2^23 or above turns into a value outside the column's range. For example, 8592090 arrives as int32(-8185126) and becomes 4286782170, the value reported in #299.

What happens on the target depends on the statement (target runs STRICT_ALL_TABLES):

  • Binlog INSERT (INSERT IGNORE): the out-of-range value is clamped to 16777215 with a warning. The data is silently corrupted and Ghostferry exits 0. Only the inline verifier notices (cutover verification failed ... PKs: 3).
  • Binlog UPDATE of an already-copied row: ERROR 1264 (22003): Out of range value and the run aborts.

The data iterator isn't affected; only values read from the binlog are.

Fix

For unsigned TYPE_MEDIUM_INT columns, keep only the low 24 bits: uint32(v) & 0xFFFFFF. This matches go-mysql's own canal fix-up (canal/rows.go). We use replication.BinlogSyncer directly, so that code never runs for us.

Why not upgrade go-mysql?

v1.15.0 (go-mysql#1117) decodes unsigned integers in replication, but only when the TABLE_MAP event carries signedness metadata (binlog_row_metadata, MySQL 8.0+). MySQL 5.7, our default test target, never sends it, so the fix-up is still needed. v1.15 also changes the MysqlGTIDSet/UUIDSet types that binlog_streamer.go builds directly. If go-mysql is upgraded later, values arrive as uint32, the type switch doesn't match, and this code does nothing.

Repro

The first commit adds the tests on their own so they can be seen failing:

  • TestBinlogUpdateEventWithUnsignedMediumInt (Go unit test, no MySQL). Before the fix: SET unsigned_col=4294967295 ... WHERE unsigned_col=4286782170.
  • test_unsigned_mediumint_with_inline_verifier (Ruby integration test). It inserts and updates rows with values ≥ 2^23 during binlog streaming. Before the fix the table checksums differ: the target holds 16777215 where the source has 8592090.

Verification (local, MySQL 8.0.32)

  • Both new tests fail on the repro commit and pass with the fix.
  • test/integration/types_test.rb: 11 tests, 0 failures.
  • go test ./test/go -run 'TestDMLEventsTestSuite|TestBinlogWriter|Types': passes.

go-mysql decodes INT24 as a sign-extended int32, so unsigned values of
2^23 and above reach the binlog writer as 4286782170-style uint32s. The
binlog INSERT IGNORE clamps them to 16777215 on the target.
@driv3r
driv3r requested a review from a team October 14, 2026 20:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

unsigned mediumint value through binlog streamer wrongfully parsed

2 participants