Export a bin1-only .gef (and .h5ad) back out of the documents #980

The idea

A light .gef written from a pyramid: everything a SAW-family reader needs, minus the extra bin levels. And an .h5ad from the cells and their labels, which is a separate and much easier job.

Two questions are deliberately still open, and this issue exists to hold them rather than to start work:

1. Is it worth building? Nobody has asked for it yet outside this conversation.
2. Does the exported file ever need to reach the cloud? If it is a local deliverable — a file you hand a collaborator — the size question below never comes up. If it must be uploaded, it is blocked on Multipart upload: single-file uploads are capped at 5 GB and fail late #978.

Why a light .gef is the right shape

Measured on the ferret hemisphere (Y00709C1E2.tissue.gef, 48.32 GB, 771,217,356 records at bin1, 26,444 genes):

Records barely shrink with bin size — 771M at bin1 against 222M at bin200, across a 40,000× area reduction — so seven levels cost roughly 4× bin1 rather than 1.3×. Dropping them is nearly all of the saving, and a reader that wants square bins can re-bin from bin1.

Ceiling: a 500 nm bin over the full chip is 60,000 bins per side if the chip is 3 × 3 cm, and the ferret already occupies ~66% of that. A fully-covered chip at the same sequencing depth is ~1.16e9 records ≈ 11.6 GB. Depth scales it linearly.

⚠️ The chip dimension needs confirming. The /wholeExp dims in the ferret's table of contents were 64,680 × 44,100, which at 500 nm is 32.3 × 22.1 mm, not 3 × 3 cm. If that is the real chip the ceiling is ~2.85e9 bins and ~9.2 GB, not 11.6. Worth settling before anyone sizes anything against it.

Fidelity: better than expected, and checkable

I initially thought a pyramid-sourced export was inherently lossy, because tiles clamp counts to uint16 and collapse (pixel, gene) pairs. Both worries are wrong here:

• The collapse cannot bite at bin1, where pixels are source pixels and nothing merges.
• Nothing clamped. ndr.format.stereoseq.readGEF reads the count wide and reports meta.nCountsClamped, and the ferret build printed "IDENTICAL — the totals STORED in the pyramid agree with the instrument vendor's own count" against SAW's /stat/gene. A single clamped count would have made those totals disagree.

So for this file the export would be exactly the records SAW wrote. An exporter should check nCountsClamped and refuse, or warn loudly, rather than assume — the guarantee is a property of the ingest, not of the format.

What has to go in besides bin1

Where it should live, and one wrinkle

Alongside buildOpossumDocuments in bscholl-genomics-python/ndiMatlabIntegration/, as a sibling harness script rather than in NDI proper until it has earned its way in.

The wrinkle: GEF wants the expression table contiguous per gene, and pyramid tiles are ordered tile → pixel → gene. So the exporter needs a re-sort by gene over ~7.7e8 records — the same memory problem makePyramid just solved, and the same fix applies (band by gene range instead of tile row).

Writing HDF5 compound datasets from MATLAB needs the low-level H5T/H5D API, perhaps 40 lines of plumbing. That is still preferable to a MATLAB→Python hop inside a batch script, so it should stay MATLAB and self-contained unless someone disagrees.

Related

• Multipart upload: single-file uploads are capped at 5 GB and fail late #978 — multipart upload. Only blocking if the answer to question 2 is "yes": ndi.cloud.uploadSingleFile does one presigned PUT, and S3 caps that at 5 GB, so a 7.7 GB export cannot be uploaded as things stand.
• The .h5ad half is independent and much easier — anndata writes it directly from cells + labels — and could land first if it is the more wanted of the two.