fix: align manifest entry schema with Java Paimon

[shyjsarah]

Purpose

Linked issue: close #741

Paimon Rust currently writes _VERSION as the last top-level field of a manifest entry. Java Paimon's versioned object format requires _VERSION at position 0, so Java manifest readers reject Rust-written manifests with:

Unexpected Manifest Avro field at position 0: expected _VERSION but found _KIND

The changelog commit path also used format identifier 0, while Java's ManifestEntrySerializer accepts identifier 2 only.

Brief change log

• Move _VERSION to the first field in ManifestEntry and MANIFEST_ENTRY_SCHEMA.
• Use format identifier 2 for both data and changelog manifest entries.
• Add a regression test for the Java-compatible top-level field order.
• Add a compatibility test proving the Rust reader can still read manifests written with the previous Rust field order.
• Add an end-to-end changelog commit assertion that reads the generated Avro manifest and verifies _VERSION = 2.

Tests

• cargo fmt --all -- --check
• cargo test -p paimon manifest_entry_ -- --nocapture
• cargo test -p paimon changelog_manifest -- --nocapture
• cargo test -p paimon table::table_commit::tests::test_commit_writes_changelog_manifest_list_size -- --nocapture
• PYO3_PYTHON=/opt/homebrew/bin/python3.11 cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
• cargo test -p paimon --all-targets --features fulltext,vortex
  • The library suite passed all 2472 executed tests.
  • Local mock REST integration tests were flaky with localhost ConnectionReset / IncompleteMessage; the initially failing token refresh test passed when rerun individually. The failures are unrelated to the manifest changes.

API and Format

No API change.

This changes newly written ManifestEntry Avro files to the canonical Java field order and writes Java's permanent format identifier 2 for changelog entries. Existing manifests written by older Paimon Rust versions remain readable by the Rust reader because decoding follows the embedded writer schema by field name.

Documentation

No documentation update is required.

[jerry-024]

One Java compatibility gap remains in the changelog path.

messages_to_changelog_entries creates every changelog ManifestEntry with version 0. These entries are then written through the same write_manifest_files path and MANIFEST_ENTRY_SCHEMA as regular data manifests.

Java also writes and reads changelog entries through the regular ManifestFile, and ManifestEntrySerializer accepts only format identifier 2. As a result, Java reading a Rust-written changelog manifest will reach checkFormatIdentifier(0) and fail with Unsupported version: 0.

Could we change this value to 2 and add a regression test covering new_changelog_files? This PR aligns the top-level field order, but changelog manifests remain incompatible until the version value is aligned as well.

[shyjsarah]

@jerry-024 Thanks for catching this. Fixed in 16da285: changelog ManifestEntry now uses format identifier 2. I also extended test_commit_writes_changelog_manifest_list_size to read the generated changelog Manifest Avro file and assert its serialized _VERSION is 2. The test reproduced Int(0) before the fix and passes after it. Related manifest/changelog tests, formatting, and workspace Clippy all pass locally.

[jerry-024]

+1