Honour the Zarr v2 order key instead of ignoring it (#95)

`.zarray`'s `order` was parsed into `ArrayMetadata.order` and then never read,
so `"order": "F"` was silently treated as row-major in both directions:

  * Reading an F-order store written by zarr-python returned the elements of
    each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7],
    [8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].

  * Writing was worse: `withOrder(Order.F)` put `"order": "F"` into `.zarray`
    while laying the chunk out row-major, so zarr-java emitted stores that
    every other implementation misreads. zarr-python read such a store as
    [[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the
    round trip through zarr-java itself returned the correct values and no
    Java-only test could see the corruption.

Zarr v3 dropped `order` and expresses the same layout with the `transpose`
codec, so `order` is modelled the same way here: a new `FortranOrderCodec`
array->array codec reverses all axes ahead of the `BytesCodec`, whose row-major
serialization of the reversed view is exactly the column-major serialization of
the logical chunk. Reversing axes is its own inverse, so one operation serves
both directions. The codec is never serialized; `order` stays the on-disk
representation, and it is only inserted for rank > 1, where the two orders
actually differ.

`decode` materializes the result with `Array.copy()`. A permuted view answers
iterators and `get(int[])` correctly but keeps a column-major backing store, so
linear accessors such as `getInt(int)` would walk it in the wrong order — and
the single-full-chunk fast path in `core.Array.read` hands a decoded chunk
straight to the caller.

Testing uses zarr-python as the oracle, since a Java-only round trip cannot
detect a symmetric bug:

  * `testdata/golden/` gains four fixtures written by zarr-python (1.3 kB in
    total, no compressor, so the chunk files are raw element bytes), covering a
    C/F pair, rank 3, and a chunk grid that leaves partial edge chunks.
    `src/test/python-scripts/generate_v2_order_golden.py` regenerates them.

  * `ZarrV2OrderTest` checks both directions against those fixtures offline,
    with no Python at test time: reads must yield `arange`, and writing
    `arange` must reproduce zarr-python's chunk bytes exactly. It also pins
    down that C and F differ on disk (uncompressed and compressed), that the
    two coincide for rank 1, that a single-full-chunk read is physically
    contiguous, and that `order` survives `resize` and attribute updates.
    7 of its 15 tests fail without this change.

  * `ZarrPythonTests` gains `testReadOrderV2`/`testWriteOrderV2`, which drive
    the installed zarr-python in both directions so that a divergence from the
    committed fixtures is noticed after a zarr-python upgrade.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: konstibob

src/main/java/dev/zarr/zarrjava/v2/Array.java

@@ -13,6 +13,7 @@
 import dev.zarr.zarrjava.utils.Utils;
 import dev.zarr.zarrjava.v2.codec.Codec;
 import dev.zarr.zarrjava.v2.codec.core.BytesCodec;
+import dev.zarr.zarrjava.v2.codec.core.FortranOrderCodec;
 
 import javax.annotation.Nonnull;
 import javax.annotation.Nullable;
@@ -36,6 +37,12 @@ protected Array(StoreHandle storeHandle, ArrayMetadata arrayMetadata) throws IOE
         this.metadata = arrayMetadata;
         this.codecPipeline = new CodecPipeline(Utils.concatArrays(
                 new Codec[]{},
+                // "order": "F" means the chunk is serialized column-major. Reversing all axes before
+                // the BytesCodec turns its row-major serialization into a column-major one. For rank
+                // 0 and 1 both orders are identical, so no codec is needed there.
+                metadata.order == Order.F && metadata.ndim() > 1
+                        ? new Codec[]{new FortranOrderCodec()}
+                        : new Codec[]{},
                 metadata.filters == null ? new Codec[]{} : metadata.filters,
                 new Codec[]{new BytesCodec(arrayMetadata.endianness.toEndian())},
                 metadata.compressor == null ? new Codec[]{} : new Codec[]{metadata.compressor}

src/main/java/dev/zarr/zarrjava/v2/codec/core/FortranOrderCodec.java

@@ -0,0 +1,102 @@
+package dev.zarr.zarrjava.v2.codec.core;
+
+import dev.zarr.zarrjava.ZarrException;
+import dev.zarr.zarrjava.core.ArrayMetadata;
+import dev.zarr.zarrjava.core.codec.ArrayArrayCodec;
+import dev.zarr.zarrjava.v2.codec.Codec;
+import ucar.ma2.Array;
+
+/**
+ * Realizes the Fortran (column-major) chunk layout that Zarr v2 selects with {@code "order": "F"}.
+ * <p>
+ * Zarr v2 stores the elements of a chunk either in row-major ({@code "C"}) or column-major
+ * ({@code "F"}) order. Zarr v3 dropped the {@code order} key and expresses the same thing with the
+ * {@code transpose} codec, so the v2 {@code order} key is modelled here the same way: as an
+ * array &rarr; array codec that reverses all axes. The row-major serialization of an array with
+ * reversed axes is exactly the column-major serialization of the original array.
+ * <p>
+ * This codec is never part of the {@code .zarray} document. It is inserted into the codec pipeline
+ * by {@link dev.zarr.zarrjava.v2.Array} whenever the metadata says {@code "order": "F"}, and the
+ * {@code order} key alone remains the on-disk representation.
+ */
+public class FortranOrderCodec extends ArrayArrayCodec implements Codec {
+
+    /**
+     * Reversing all axes is its own inverse, so encoding and decoding differ only in what they do
+     * with the resulting view.
+     */
+    private Array reverseAxes(Array chunkArray) throws ZarrException {
+        int ndim = resolveArrayMetadata().ndim();
+        if (chunkArray.getRank() != ndim) {
+            throw new ZarrException(
+                    "Chunk array has rank " + chunkArray.getRank() + ", but the array has " + ndim
+                            + " dimensions.");
+        }
+        int[] reversedAxes = new int[ndim];
+        for (int i = 0; i < ndim; i++) {
+            reversedAxes[i] = ndim - 1 - i;
+        }
+        return chunkArray.permute(reversedAxes);
+    }
+
+    /**
+     * Turns the logical chunk into the axis-reversed view that the following
+     * {@link dev.zarr.zarrjava.core.codec.core.BytesCodec} then serializes row-major, which yields
+     * the column-major serialization of the logical chunk. The view is handed on as-is: the
+     * {@code BytesCodec} flattens in logical index order, so no copy is needed here.
+     */
+    @Override
+    public Array encode(Array chunkArray) throws ZarrException {
+        return reverseAxes(chunkArray);
+    }
+
+    /**
+     * Turns the axis-reversed array that came out of the
+     * {@link dev.zarr.zarrjava.core.codec.core.BytesCodec} back into the logical chunk.
+     * <p>
+     * The result is materialized with {@link Array#copy()} rather than returned as a permuted view.
+     * A view's logical order is correct for iterators and index-based access, but its backing store
+     * is still in column-major order, so linear accessors such as {@code getInt(int)} would return
+     * elements in the wrong order. Chunks decoded here can be handed straight to callers by the
+     * single-full-chunk fast path in {@link dev.zarr.zarrjava.core.Array#read(long[], long[])}, so
+     * they have to behave like any other decoded chunk.
+     */
+    @Override
+    public Array decode(Array chunkArray) throws ZarrException {
+        return reverseAxes(chunkArray).copy();
+    }
+
+    /**
+     * Reports the axis-reversed shapes to the following codec, so that the
+     * {@link dev.zarr.zarrjava.core.codec.core.BytesCodec} reads and writes the chunk bytes with the
+     * reversed chunk shape.
+     */
+    @Override
+    public ArrayMetadata.CoreArrayMetadata resolveArrayMetadata() throws ZarrException {
+        super.resolveArrayMetadata();
+        int ndim = arrayMetadata.ndim();
+        long[] reversedShape = new long[ndim];
+        int[] reversedChunkShape = new int[ndim];
+        for (int i = 0; i < ndim; i++) {
+            reversedShape[i] = arrayMetadata.shape[ndim - 1 - i];
+            reversedChunkShape[i] = arrayMetadata.chunkShape[ndim - 1 - i];
+        }
+        return new ArrayMetadata.CoreArrayMetadata(
+                reversedShape,
+                reversedChunkShape,
+                arrayMetadata.dataType,
+                arrayMetadata.parsedFillValue
+        );
+    }
+
+    @Override
+    public long computeEncodedSize(long inputByteLength,
+                                   ArrayMetadata.CoreArrayMetadata arrayMetadata) {
+        return inputByteLength;
+    }
+
+    @Override
+    public Codec evolveFromCoreArrayMetadata(ArrayMetadata.CoreArrayMetadata arrayMetadata) {
+        return this;
+    }
+}

src/test/java/dev/zarr/zarrjava/ZarrPythonTests.java

@@ -360,4 +360,82 @@ public void testGroupReadWriteV3() throws Exception {
         Assertions.assertArrayEquals(new int[]{16, 16, 16}, result.getShape());
         assertIsTestdata(result, dataType);
     }
+
+    /**
+     * The Zarr v2 {@code order} cases, as (shape, chunks, order).
+     */
+    static Stream<Object[]> orderProviderV2() {
+        return Stream.of(
+                new Object[]{"3,4", "3,4", "C"},
+                new Object[]{"3,4", "3,4", "F"},
+                new Object[]{"2,3,4", "2,3,4", "F"},
+                // Chunks divide neither dimension evenly, so the trailing chunks are partial.
+                new Object[]{"5,7", "2,3", "F"},
+                new Object[]{"5,7,3", "2,3,2", "F"},
+                // For rank 1 both orders describe the same layout.
+                new Object[]{"10", "4", "F"}
+        );
+    }
+
+    /**
+     * Live counterpart to {@link ZarrV2OrderTest}, which covers the same ground offline against
+     * committed fixtures. This runs against the installed zarr-python so that a change in the
+     * reference implementation is noticed rather than silently diverging from the fixtures.
+     */
+    @ParameterizedTest
+    @MethodSource("orderProviderV2")
+    public void testReadOrderV2(String shape, String chunks, String order) throws Exception {
+        StoreHandle storeHandle = new FilesystemStore(TESTOUTPUT)
+                .resolve("testReadOrderV2", shape + "_" + chunks + "_" + order);
+        run_python_script("zarr_python_order_v2.py", "write", storeHandle.toPath().toString(),
+                shape, chunks, order);
+
+        dev.zarr.zarrjava.v2.Array array = dev.zarr.zarrjava.v2.Array.open(storeHandle);
+        ucar.ma2.Array result = array.read();
+
+        Assertions.assertEquals(dev.zarr.zarrjava.v2.Order.valueOf(order), array.metadata().order);
+        for (int i = 0; i < result.getSize(); i++) {
+            Assertions.assertEquals(i, result.getInt(i),
+                    "element " + i + " of a " + order + "-order array");
+        }
+    }
+
+    /**
+     * The write direction, verified by zarr-python. zarr-java used to write {@code "order": "F"}
+     * into {@code .zarray} while laying the chunk out row-major, which its own reader then read back
+     * correctly &mdash; so only an external reader can catch it.
+     */
+    @ParameterizedTest
+    @MethodSource("orderProviderV2")
+    public void testWriteOrderV2(String shape, String chunks, String order) throws Exception {
+        StoreHandle storeHandle = new FilesystemStore(TESTOUTPUT)
+                .resolve("testWriteOrderV2", shape + "_" + chunks + "_" + order);
+
+        long[] shapeArray = Arrays.stream(shape.split(",")).mapToLong(Long::parseLong).toArray();
+        int[] chunksArray = Arrays.stream(chunks.split(",")).mapToInt(Integer::parseInt).toArray();
+
+        dev.zarr.zarrjava.v2.Array array = dev.zarr.zarrjava.v2.Array.create(
+                storeHandle,
+                dev.zarr.zarrjava.v2.Array.metadataBuilder()
+                        .withShape(shapeArray)
+                        .withChunks(chunksArray)
+                        .withDataType(dev.zarr.zarrjava.v2.DataType.INT32)
+                        .withOrder(dev.zarr.zarrjava.v2.Order.valueOf(order))
+                        .withFillValue(0)
+                        .build()
+        );
+
+        int[] intShape = new int[shapeArray.length];
+        for (int i = 0; i < shapeArray.length; i++) {
+            intShape[i] = (int) shapeArray[i];
+        }
+        ucar.ma2.Array data = ucar.ma2.Array.factory(ucar.ma2.DataType.INT, intShape);
+        for (int i = 0; i < data.getSize(); i++) {
+            data.setInt(i, i);
+        }
+        array.write(new long[shapeArray.length], data);
+
+        run_python_script("zarr_python_order_v2.py", "read", storeHandle.toPath().toString(),
+                shape, chunks, order);
+    }
 }