Skip to content

[blk storage] Adopt the Orderer optimizations for the block storage #167

Open
1 of 3 issues completed
Open

Description

@liran-funaro

The Fabric-X Orderer carries its own copy of this package at
common/ledger/blkstorage.
It is a sibling fork, not a descendant — both trees come from Hyperledger Fabric's blkstorage, and
each has diverged since. cache.go, block_stream.go, config.go, metrics.go and
protobuf_util.go are still byte-identical; the rest differs in both directions.

The divergence that matters here has one cause. All three orderer ledgers — assembler, batcher, and
consenter — configure their store with AttrsToIndex: []IndexableAttr{IndexableAttrBlockNum} and
nothing else, so no transaction ID index exists anywhere in the orderer. The same is true of a
Fabric-X Committer sidecar with the transaction ID index disabled
(hyperledger/fabric-x-committer#784). Both therefore pay for per-transaction index information that
nothing ever reads. The orderer's fork stopped paying; this repo's copy still does.

This umbrella tracks porting those wins back. Three, in descending order of measured impact:

  1. [blkstorage] Do not build tx index information no index will read #165 — do not build transaction index information no index will read, on the write path
    (serializeBlock). Addressed by [blkstorage] Do not build tx index information no index will read #166, which is a superset of the orderer's version: it derives the
    decision from the index configuration rather than a caller-passed bool, also skips the offsets and
    not just the transaction IDs, and pre-sizes the serialization buffer. Nothing left to port.
  2. [blkstorage] Do not compute transaction IDs when deserializing a block #182 — the read path — extractData computes a transaction ID per envelope on every block read and
    deserializeBlock discards all of it. Not touched by [blkstorage] Do not build tx index information no index will read #166. This is the same unmarshal as [blkstorage] Do not build tx index information no index will read #165, on
    the other side of the store, and unlike [blkstorage] Do not build tx index information no index will read #165 it is wasted unconditionally rather than only for
    stores without a transaction ID index.
  3. [blkstorage] Extract only the header when only the header is read #183 — header-only reads — retrieveBlockHeaderByNumber and retrieveFirstBlockNumFromFile parse
    every envelope and all metadata to read a three-field header.

Not in scope, and not to be adopted: the orderer's fork dropped snapshot bootstrap and export,
reset and rollback, and left TODOs where the snapshot front-boundary guards used to sit,
marking them for ledger pruning. This repo is also ahead of the orderer on AddBlockNoSync/Flush
with a sync before file rollover (#69), the context-aware blocksItr (#88), the ErrNotFound
sentinel (#90), and cockroachdb/errors. Each item below is therefore additive to this tree — there
is no orderer behaviour to take wholesale.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestperformancePerformance improvement or regression

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions