馃 Follow-up to #152: converting NaNs to Arrow nulls is the correct default for SQL aggregation semantics, but some projection-only consumers already carry an explicit validity mask and need the original float payload rather than SQL null semantics.
For example, Samudra selects complete ocean-state planes for ML batches and applies its own wet mask. iter_record_batches() currently uses pa.array(..., from_pandas=True) for every projected variable, which scans for NaNs and constructs validity information even though the query only filters on dimension/index columns.
Would an opt-in registration/read mode such as nan_as_null=False be reasonable? The default must remain True so AVG, COUNT, IS NULL, etc. preserve the behavior fixed by #152. The opt-in mode should preserve NaN as an ordinary Arrow float and document that SQL null operations will not treat it as missing.
This should be benchmarked before implementation; the goal is to determine whether skipping null discovery/bitmap construction materially improves projection-heavy dense-array scans and reduces memory.
馃 Follow-up to #152: converting NaNs to Arrow nulls is the correct default for SQL aggregation semantics, but some projection-only consumers already carry an explicit validity mask and need the original float payload rather than SQL null semantics.
For example, Samudra selects complete ocean-state planes for ML batches and applies its own wet mask.
iter_record_batches()currently usespa.array(..., from_pandas=True)for every projected variable, which scans for NaNs and constructs validity information even though the query only filters on dimension/index columns.Would an opt-in registration/read mode such as
nan_as_null=Falsebe reasonable? The default must remainTruesoAVG,COUNT,IS NULL, etc. preserve the behavior fixed by #152. The opt-in mode should preserve NaN as an ordinary Arrow float and document that SQL null operations will not treat it as missing.This should be benchmarked before implementation; the goal is to determine whether skipping null discovery/bitmap construction materially improves projection-heavy dense-array scans and reduces memory.