Search before asking
Motivation
When processing a KV write batch, KvTablet may need to read the existing value of each key to generate the correct merged value and CDC records.
Currently, the write path checks the pre-write buffer first and then performs an individual RocksDB point lookup for every missing key. For a batch containing N distinct keys whose values are stored in RocksDB, this can result in N RocksDB JNI calls.
These lookups are executed while the tablet write lock is held. The repeated JNI crossings and serialized point lookups therefore increase the lock-holding time and limit write throughput, especially for large partial-update batches and tables using the FULL changelog image.
Solution
atch the RocksDB old-value lookups performed while processing one KV write batch:
- Decode the incoming records and identify the keys that require an old value.
- Check the pre-write buffer first because it contains values newer than RocksDB.
- Preserve pre-write tombstones without falling back to RocksDB.
- Deduplicate the remaining keys and load them with one RocksDB
multiGet.
- Continue processing records in their original order.
- Recheck the pre-write buffer before using a prefetched RocksDB value so that repeated keys in the same batch observe earlier mutations from that batch.
- Keep the existing fast path for WAL changelog writes that do not require old values.
This optimization should reduce RocksDB JNI crossings for old-value lookup from up to N point-get calls to at most one multi-get call per KV record batch.
The change must not alter:
- KV tablet locking semantics.
- Pre-write buffer visibility.
- Record processing order.
- WAL and CDC ordering.
- Partial-update, aggregation, versioned, first-row, or auto-increment semantics.
- Duplicate batch detection and rollback behavior.
- Flush, snapshot, recovery, or lifecycle behavior.
Anything else?
No response
Willingness to contribute
Search before asking
Motivation
When processing a KV write batch,
KvTabletmay need to read the existing value of each key to generate the correct merged value and CDC records.Currently, the write path checks the pre-write buffer first and then performs an individual RocksDB point lookup for every missing key. For a batch containing N distinct keys whose values are stored in RocksDB, this can result in N RocksDB JNI calls.
These lookups are executed while the tablet write lock is held. The repeated JNI crossings and serialized point lookups therefore increase the lock-holding time and limit write throughput, especially for large partial-update batches and tables using the FULL changelog image.
Solution
atch the RocksDB old-value lookups performed while processing one KV write batch:
multiGet.This optimization should reduce RocksDB JNI crossings for old-value lookup from up to N point-get calls to at most one multi-get call per KV record batch.
The change must not alter:
Anything else?
No response
Willingness to contribute