Skip to content

Ignore stale InlineVerifier coordinate when resuming without the verifier - #464

Open
driv3r wants to merge 2 commits into
mainfrom
fix-resume-without-inline-verifier
Open

driv3r wants to merge 2 commits into
mainfrom
fix-resume-without-inline-verifier

Conversation

@driv3r

@driv3r driv3r commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Fixes #184. Supersedes #185 (same approach by @ckolbitsch-work, ported to the BinlogCoordinate API so it covers GTID mode too).

Problem

On resume, Ferry.Start always connects the source streamer at SerializableState.MinSourceBinlogCoordinate(). In file mode that is the older of the binlog writer and InlineVerifier positions; in GTID mode it is the intersection of the two sets.

The InlineVerifier coordinate only advances while the verifier runs (ferry.go skips the startup update when f.inlineVerifier == nil). It is still copied into every later state, though. So with a state written by a verifier-enabled run, every later resume without the verifier goes back to that old coordinate. Once the source purges those binlogs, resuming fails, even though the binlog writer coordinate is still available:

  • file/position: ERROR 1236 (HY000): Could not find first log file name in binary log index file
  • GTID: ERROR 1236 (HY000): Cannot replicate because the master purged required binary logs

Fix

When the InlineVerifier is disabled, resume from WrittenSourceBinlogCoordinate() instead of MinSourceBinlogCoordinate().

The InlineVerifier coordinate stays in the state untouched, as discussed in #185: if the verifier is re-enabled later, the resume point still includes it (or fails loudly if those binlogs are gone). Clearing it would quietly drop that coverage.

Repro

The first commit adds the test on its own: test_interrupt_resume_without_inline_verifier_ignores_stale_inline_verifier_coordinate.

  1. Run with verifier_type: "Inline" and interrupt. The state now has an InlineVerifier coordinate.
  2. Insert a row on the source, then FLUSH BINARY LOGS.
  3. Resume without a verifier, insert another row, wait for it to reach the target, and interrupt. The writer coordinate is now past the rotation; the InlineVerifier coordinate is unchanged (asserted).
  4. PURGE BINARY LOGS TO the rotated file.
  5. Resume without a verifier and assert source and target match.

Before the fix, step 5 exits with the ERROR 1236 messages above in both coordinate modes. The test also moves the SHOW MASTER STATUS / SHOW BINARY LOG STATUS version switch into a source_binlog_status helper, which test_interrupt_resume_between_consecutive_rows_events now uses as well.

Verification (local, MySQL 8.0.32)

  • New test: fails on the repro commit and passes with the fix, in file_position and gtid modes.
  • test/integration/interrupt_resume_test.rb: 26 tests, 0 failures, in both modes.
  • test/integration/inline_verifier_test.rb: 22 tests, 0 failures, in both modes.
  • go test ./test/go -run 'TestSerializableState|TestStateTracker|TestBinlogCoordinate' and the root-package GTID/resume tests: pass.

Known gap (not changed here)

With the verifier disabled, the state dump drops BinlogVerifyStore. So rows still waiting for re-verification when the verifier was turned off are lost if it is re-enabled later. That behaviour is the same before and after this PR.

A state written by an InlineVerifier run keeps its inline-verifier
coordinate when later runs resume without the verifier. Once the source
purges the binlogs it points at, resuming fails with ERROR 1236 even
though the binlog writer coordinate is still available.
@driv3r
driv3r requested a review from a team October 14, 2026 20:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resuming ghostferry may fail due to inactive verifier

2 participants