Consus

Pure-Rust, no_std-compatible, memory-safe reimplementation of hierarchical and array-oriented storage formats for scientific computing.

Overview

Consus replaces C-dependent bindings (hdf5-rs, netCDF-sys, etc.) with a native Rust implementation providing:

• Hyperslab and selection reads over a positioned-read (ReadAt) abstraction
• Full compression support (zlib, gzip, zstd, lz4, blosc, szip)
• Thread-safe parallel I/O through Moirai, sized from Themis CPU topology
• WASM and embedded targets (no_std compatible core)
• Pluggable backend architecture for format interoperability

I/O and copy behavior

State of the tree, not a roadmap:

• Positioned reads (ReadAt::read_at) fill a caller-provided buffer. Hyperslab and selection reads materialize an owned Vec<u8> per region.
• MmapReader::as_slice (consus-io) exposes the memory mapping directly, so callers that want to read a mapped file without a copy can borrow from it.
• consus-onnx parses borrowed: ModelDocument<'a>, Initializer<'a>, and Node<'a> alias the source protobuf buffer rather than copying names and initializer payloads.
• The ZeroCopyRead trait (consus::sync) is not implemented: its single blanket impl<T: ReadAt> always allocates and returns ByteView::Owned, so is_zero_copy() is false for every source in the tree. Because it is a blanket impl, a per-type borrowing implementation (for example on MmapReader) would also be a coherence conflict. Treat the trait as reserved surface, not a capability.

Supported Formats

The shared raster boundary is defined in ADR 0004. consus-raster owns JPEG integer samples and bounded EXIF interpretation for RITK and Metis. Medical modality conversion and display placement stay with those consumers. Migration acceptance is tracked by CONSUS-RASTER-001.

Format	Status	Spec Compliance
HDF5	Phase 1 – Complete	Full read and write paths implemented (v1/v2/v3 superblocks, all datatype classes, multi-chunk with filter pipeline). Chunked dataset write with layout v3/v4 and async I/O paths verified.
FITS	Phase 2 – Complete	Full read/write for primary images, IMAGE extensions, ASCII tables, and binary tables
Zarr v2/v3	Phase 2 – Complete	Metadata parsing, codec pipeline, chunk read/write, full-array read/write, partial selection read/write, Python fixture-backed interoperability for v2 and v3 stores; boundary-chunk stride correction and sharding codec verified; v3 metadata write path preserves dimension names and group attributes.
MATLAB .mat	Phase 2 – Complete	v4 (binary, all precisions), v5 (binary, all mxClass types including cell/struct/sparse, miCOMPRESSED), v7.3 (HDF5-backed, numeric/logical/char/struct/cell, deterministic cell ordering, scalar struct decoding)
netCDF-4	Phase 2 – Complete	HDF5-backed semantic extraction implemented for dimension scales, variables, groups, decoded attributes, unlimited-dimension propagation, ancestor-scope dimension resolution for nested groups, DIMENSION_LIST-based variable-to-dimension binding. Classic and enhanced model read and write paths are fully implemented and verified.
Apache Parquet	Phase 3 – Complete	Canonical schema mapping implemented; validated in-memory dataset descriptor, row-group metadata, ordered column projection model, and value-semantic tests implemented. Nested group fields map to canonical compound datatypes and repeated fields map to canonical variable-length datatypes. Thrift footer decoding, file-backed Parquet reader, and full wire-level write path are complete and verified.
NWB (Neurodata Without Borders)	Phase 3 – Complete	Complete HDF5-backed read and write compliance for TimeSeries, Units, ElectrodeTable, and Subject models. End-to-end NWB 2.x conformance validation engine implemented.
NumPy NPY/NPZ	Phase 3 – Typed core complete	Typed little-endian f32, f64, i32, and i64 arrays; validated shape/header parsing; ZIP-backed named arrays; no ndarray dependency.
ONNX	Phase 3 – Document core complete	Bounded zero-copy protobuf decoding for graph topology, tensor metadata, operator sets, and borrowed initializer payloads without a tensor-runtime dependency.

Architecture

consus (facade)
├── consus-core        # Core types, traits, error types (no_std)
├── consus-io          # Sync/async I/O abstractions
├── consus-compression # Codec registry (zlib, zstd, lz4, etc.)
├── consus-hdf5        # HDF5 format implementation
├── consus-zarr        # Zarr v2/v3 implementation
├── consus-netcdf      # netCDF-4 semantic model + HDF5 mapping layer
├── consus-arrow       # Arrow semantic model and bridge planning layer
├── consus-fits        # FITS format implementation
├── consus-mat         # MATLAB .mat v4/v5/v7.3 reader
├── consus-hdmf        # HDMF DynamicTable read/write
├── consus-nwb         # Neurodata Without Borders models over HDF5/HDMF
├── consus-npy         # Typed NPY arrays and NPZ archives
├── consus-zip         # Bounded ZIP container (NPZ, 3MF)
├── consus-onnx        # Bounded zero-copy ONNX document decoding
└── consus-parquet     # Parquet interop layer

consus-raster          # JPEG samples and bounded EXIF metadata
consus-python          # PyO3 extension module (wheel-only, publish = false)

All eighteen crates above are [workspace] members of the root Cargo.toml; tests is an additional non-publishable integration-test member.

Quick Start

use consus::File;

fn main() -> consus::Result<()> {
    // Create a new HDF5 file
    let file = File::create("experiment.h5")?;
    let group = file.create_group("/simulations/run_001")?;

    let data: Vec<f64> = vec![1.0, 2.0, 3.0, 4.0];
    group.create_dataset("temperature")
        .shape(&[2, 2])
        .write(&data)?;

    Ok(())
}

Python Distribution

The Python extension is distributed as consus-python and imported as consus. GitHub Releases tagged consus-python-v<version> produce locked Linux, Windows, and universal macOS wheels, attach the validated wheel set and SHA-256 checksums to the release, and publish the same artifacts to PyPI through OIDC Trusted Publishing. Python package metadata reads the consus-python Cargo version rather than maintaining a second version field.

Rust Crate Releases

The Crates.io Release workflow validates a named workspace package on manual dispatch. After that package's required first release is published locally and its crates.io Trusted Publisher is registered, a GitHub Release tagged crate-<package>-v<version> packages, verifies, and publishes the matching Cargo version with a short-lived OIDC token. Validation runs in a separate read-only job. The publish job is bound to the GitHub crates-io environment; register each package's Trusted Publisher with that environment. Python bindings remain a wheel-only artifact and are marked publish = false for crates.io.

Current HDF5 Verification Status

Current repository verification indicates:

• HDF5 read support covers superblocks, object headers, datatype parsing, dataspace parsing, link traversal, attribute parsing, contiguous dataset reads, chunk metadata parsing, dense link and dense attribute enumeration, and soft-link path resolution.
• HDF5 write support currently covers superblock v2 writing, object header v2 writing, datatype/dataspace/layout encoding, contiguous dataset data blocks, hard-link encoding, soft-link encoding, and attribute encoding.
• Chunked dataset write is implemented: the v3 data layout message, v1 raw-data chunk B-tree leaf index, resolved chunk index address, and filter pipeline metadata are all serialized correctly. End-to-end value roundtrip is verified by chunked_dataset_value_roundtrip.
• Compressed chunked writes are tracked under the filter pipeline; full compression roundtrip coverage is in progress.

Target Users

• Research labs and simulation teams
• ML pipelines requiring reproducible data storage
• Climate, geospatial, and bioinformatics workflows
• Enterprise R&D requiring long-term data preservation
• Embedded and WASM deployments needing portable I/O

Strategic Advantages

• Portability: Single cargo add consus — no system libraries, no CMake, no pkg-config
• Auditability: Pure Rust source, no C FFI boundary to audit
• Supply-chain security: Minimal, auditable dependency tree
• Cross-compilation: Targets WASM, ARM, RISC-V without toolchain pain
• Memory safety: two unsafe blocks in the whole workspace — the memmap2 mapping call in consus-io/src/io/sync/mmap.rs and one slice reinterpretation in consus/src/builders/dataset.rs, both carrying // SAFETY: comments. No format parser or encoder contains unsafe. This is currently a property of the code rather than a compiler-enforced one: only consus-hdmf, consus-npy and consus-zip declare #![forbid(unsafe_code)]
• Astronomy pipeline interoperability: FITS support enables direct ingestion of telescope image products, calibration frames, and catalog tables into the same unified API used for other scientific formats
• Spectral and survey R&D continuity: One storage facade now spans astronomy archive interchange and downstream analysis workflows, reducing format-specific glue code in spectral reduction, sky survey processing, and observatory data engineering

Minimum Supported Rust Version

1.95.0 (edition 2024)

Pre-push hook

Install the hooks once per clone:

git config core.hooksPath .githooks

Git never applies tracked hooks on its own, so this is a one-time step per clone. The pre-push hook runs scripts/lockfile.py --check, which is the same check CI runs. It matters most when working inside the Atlas stack: the stack's [patch] overlay makes cargo resolve first-party dependencies to local paths and write a Cargo.lock with every source = "git+..." line stripped. That lock resolves fine under the overlay and fails every --locked job in CI, so without the hook the corruption is invisible until a runner reports it. Repair with python3 scripts/lockfile.py --regenerate.

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT License (LICENSE-MIT)

at your option.