馃 Samudra's canonical data-loader experiment found that ORDER BY time, lat, lon is a material part of point-read latency even though XarrayScanExec already emits each dense block in C-order.
Local benchmark setup: an uncompressed float32 Zarr store with 32 times, 16 variables, a 180脳360 grid, chunks=None at Xarray open, and XQL scan partitions of {"time": 1}. Selecting four times and all 16 variables took approximately:
- ordered DataFusion collection: 21 ms
- the same query without
ORDER BY: 13 ms
The ordered physical plan contains:
SortPreservingMergeExec: time, lat, lon
SortExec: time, lat, lon
FilterExec: time IN (...)
RepartitionExec
XarrayScanExec
For an ML loader, relying on unspecified result order is not acceptable, and scattering unordered rows in NumPy erased the saved time. Could the native provider declare the scan's natural intra-partition ordering in PlanProperties, plus enough partition-range ordering for DataFusion to use a preserving merge instead of sorting the cells?
A useful regression test would assert both exact ordered results and the absence of SortExec in EXPLAIN for an order compatible with the dataset's dimension order. This may also reduce sort-buffer memory for dense-grid round trips.
馃 Samudra's canonical data-loader experiment found that
ORDER BY time, lat, lonis a material part of point-read latency even thoughXarrayScanExecalready emits each dense block in C-order.Local benchmark setup: an uncompressed float32 Zarr store with 32 times, 16 variables, a 180脳360 grid,
chunks=Noneat Xarray open, and XQL scan partitions of{"time": 1}. Selecting four times and all 16 variables took approximately:ORDER BY: 13 msThe ordered physical plan contains:
For an ML loader, relying on unspecified result order is not acceptable, and scattering unordered rows in NumPy erased the saved time. Could the native provider declare the scan's natural intra-partition ordering in
PlanProperties, plus enough partition-range ordering for DataFusion to use a preserving merge instead of sorting the cells?A useful regression test would assert both exact ordered results and the absence of
SortExecinEXPLAINfor an order compatible with the dataset's dimension order. This may also reduce sort-buffer memory for dense-grid round trips.