Skip to content

Add single ALP test file (216KB, covers corner cases) - #119

Open
alamb wants to merge 5 commits into
apache:masterfrom
alamb:alamb/alp_test_data
Open

Add single ALP test file (216KB, covers corner cases)#119
alamb wants to merge 5 commits into
apache:masterfrom
alamb:alamb/alp_test_data

Conversation

@alamb

@alamb alamb commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

(Draft until I validate this example some more)

Rationale

#100 from @prtkgaur and @vinooganesh at prtkgaur#1 have coverage but are also 4x larger than all the existing data files

Thus I want to create an example file that can be used by readers but is not large and still covers all the corner cases

Changes

  1. Adds a single 216 KB file created with a CLI driver based on the C++ implementation
  1. Describes the contents and rationale in data/README.md. I found the rendered preview easier to read: https://github.com/alamb/parquet-testing/blob/alamb/alp_test_data/data/README.md#alp-encoding
full file layout details

Wrote /tmp/alp_extended.zstd.parquet (218992 bytes)

File: /tmp/alp_extended.zstd.parquet
  rows: 9032, row groups: 5, columns: 8

Column 'float_plain' (FLOAT)
  Row group 0: rows=6144 encodings=[ RLE PLAIN ] compression=zstd null_count=0 compressed_bytes=21005
  Row group 1: rows=1024 encodings=[ RLE PLAIN ] compression=zstd null_count=0 compressed_bytes=3863
  Row group 2: rows=1024 encodings=[ RLE PLAIN ] compression=zstd null_count=0 compressed_bytes=47
  Row group 3: rows=808 encodings=[ RLE PLAIN ] compression=zstd null_count=8 compressed_bytes=2672
  Row group 4: rows=32 encodings=[ RLE PLAIN ] compression=zstd null_count=0 compressed_bytes=162

Column 'double_plain' (DOUBLE)
  Row group 0: rows=6144 encodings=[ RLE PLAIN ] compression=zstd null_count=0 compressed_bytes=24880
  Row group 1: rows=1024 encodings=[ RLE PLAIN ] compression=zstd null_count=0 compressed_bytes=5882
  Row group 2: rows=1024 encodings=[ RLE PLAIN ] compression=zstd null_count=0 compressed_bytes=53
  Row group 3: rows=808 encodings=[ RLE PLAIN ] compression=zstd null_count=8 compressed_bytes=2376
  Row group 4: rows=32 encodings=[ RLE PLAIN ] compression=zstd null_count=0 compressed_bytes=291

Column 'float_alp_1024' (FLOAT)
  Row group 0: rows=6144 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=16430
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=10 (vector_size=1024) num_elements=6144
    num_vectors=6 total_exceptions=1546
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      1024      6   4   11  4294966296           0     
    1      1024      6   4   11  4294966296           7      0:nan, 476:nan, 976:inf, 977:-inf, 978:-0, ...
    2      1024      6   4   11  4294966297           1      452:3.1415927410125732
    3      1024      6   4   11  4294966296           2      28:44974936064, 639:-1243432448
    4      1024      6   4   11  4294966296           512    0:-5.6894025802612305, 2:-5.7471122741699219, 4:5.2631058692932129, 6:3.063255786895752, 8:7.4240894317626953, ...
    5      1024      6   4   0   0                    1024   0:0.98619550466537476, 1:-3.7603194713592529, 2:-0.34999150037765503, 3:0.1334528774023056, 4:-2.2552545070648193, ...
  Row group 1: rows=1024 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=2351
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=10 (vector_size=1024) num_elements=1024
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      1024      6   2   18  4294867362           0     
  Row group 2: rows=1024 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=45
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=10 (vector_size=1024) num_elements=1024
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      1024      10  8   0   777                  0     
  Row group 3: rows=808 encodings=[ RLE ALP ] compression=uncompressed null_count=8 compressed_bytes=1186
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=10 (vector_size=1024) num_elements=800
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      800       6   4   11  4294966297           0     
  Row group 4: rows=32 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=237
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=10 (vector_size=1024) num_elements=32
    num_vectors=1 total_exceptions=32
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        10  10  0   0                    32     0:-7.9999998744539955e+18, 1:7.9999998744539955e+18, 2:3.2175613495004365e+18, 3:-1.428295291294974e+18, 4:74196599420485632, ...

Column 'double_alp_1024' (DOUBLE)
  Row group 0: rows=6144 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=22638
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=10 (vector_size=1024) num_elements=6144
    num_vectors=6 total_exceptions=1546
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      1024      14  12  11  18446744073709550616 0     
    1      1024      14  12  11  18446744073709550616 7      0:nan, 476:nan, 976:inf, 977:-inf, 978:-0, ...
    2      1024      14  12  11  18446744073709550617 1      452:3.1415926535897931
    3      1024      14  12  11  18446744073709550616 2      28:44974934523.343002, 639:-1243432432.3432
    4      1024      14  12  11  18446744073709550616 512    0:-5.6894026020883821, 2:-5.7471123414096521, 4:5.2631058917995048, 6:3.0632557599260757, 8:7.4240896660071236, ...
    5      1024      14  12  0   0                    1024   0:0.98619550441899628, 1:-3.7603195733564965, 2:-0.34999150139969903, 3:0.13345287723017574, 4:-2.2552545378626987, ...
  Row group 1: rows=1024 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=2355
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=10 (vector_size=1024) num_elements=1024
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      1024      16  12  18  18446744073709451682 0     
  Row group 2: rows=1024 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=49
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=10 (vector_size=1024) num_elements=1024
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      1024      16  14  0   777                  0     
  Row group 3: rows=808 encodings=[ RLE ALP ] compression=uncompressed null_count=8 compressed_bytes=1190
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=10 (vector_size=1024) num_elements=800
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      800       14  12  11  18446744073709550617 0     
  Row group 4: rows=32 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=305
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=10 (vector_size=1024) num_elements=32
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        0   0   64  10446744073709551616 0     

Column 'float_alp_4096' (FLOAT)
  Row group 0: rows=6144 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=17786
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=12 (vector_size=4096) num_elements=6144
    num_vectors=2 total_exceptions=1546
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      4096      6   4   11  4294966296           10     1024:nan, 1500:nan, 2000:inf, 2001:-inf, 2002:-0, ...
    1      2048      6   4   11  4294966296           1536   0:-5.6894025802612305, 2:-5.7471122741699219, 4:5.2631058692932129, 6:3.063255786895752, 8:7.4240894317626953, ...
  Row group 1: rows=1024 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=2351
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=12 (vector_size=4096) num_elements=1024
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      1024      6   2   18  4294867362           0     
  Row group 2: rows=1024 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=45
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=12 (vector_size=4096) num_elements=1024
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      1024      10  8   0   777                  0     
  Row group 3: rows=808 encodings=[ RLE ALP ] compression=uncompressed null_count=8 compressed_bytes=1186
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=12 (vector_size=4096) num_elements=800
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      800       6   4   11  4294966297           0     
  Row group 4: rows=32 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=237
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=12 (vector_size=4096) num_elements=32
    num_vectors=1 total_exceptions=32
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        10  10  0   0                    32     0:-7.9999998744539955e+18, 1:7.9999998744539955e+18, 2:3.2175613495004365e+18, 3:-1.428295291294974e+18, 4:74196599420485632, ...

Column 'double_alp_4096' (DOUBLE)
  Row group 0: rows=6144 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=23978
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=12 (vector_size=4096) num_elements=6144
    num_vectors=2 total_exceptions=1546
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      4096      14  12  11  18446744073709550616 10     1024:nan, 1500:nan, 2000:inf, 2001:-inf, 2002:-0, ...
    1      2048      14  12  11  18446744073709550616 1536   0:-5.6894026020883821, 2:-5.7471123414096521, 4:5.2631058917995048, 6:3.0632557599260757, 8:7.4240896660071236, ...
  Row group 1: rows=1024 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=2355
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=12 (vector_size=4096) num_elements=1024
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      1024      16  12  18  18446744073709451682 0     
  Row group 2: rows=1024 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=49
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=12 (vector_size=4096) num_elements=1024
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      1024      16  14  0   777                  0     
  Row group 3: rows=808 encodings=[ RLE ALP ] compression=uncompressed null_count=8 compressed_bytes=1190
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=12 (vector_size=4096) num_elements=800
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      800       14  12  11  18446744073709550617 0     
  Row group 4: rows=32 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=305
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=12 (vector_size=4096) num_elements=32
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        0   0   64  10446744073709551616 0     

Column 'float_alp_32' (FLOAT)
  Row group 0: rows=6144 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=18848
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=5 (vector_size=32) num_elements=6144
    num_vectors=192 total_exceptions=1546
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        6   4   11  4294966383           0     
    1      32        6   4   11  4294966316           0     
    2      32        6   4   11  4294966309           0     
    3      32        6   4   11  4294966310           0     
    4      32        6   4   11  4294966336           0     
    5      32        6   4   11  4294966502           0     
    6      32        6   4   11  4294966391           0     
    7      32        6   4   11  4294966485           0     
    8      32        6   4   11  4294966297           0     
    9      32        6   4   11  4294966418           0     
    10     32        6   4   11  4294966439           0     
    11     32        6   4   11  4294966297           0     
    12     32        6   4   11  4294966448           0     
    13     32        6   4   11  4294966312           0     
    14     32        6   4   11  4294966313           0     
    15     32        6   4   11  4294966450           0     
    16     32        6   4   11  4294966315           0     
    17     32        6   4   11  4294966300           0     
    18     32        6   4   11  4294966412           0     
    19     32        6   4   11  4294966389           0     
    20     32        6   4   11  4294966318           0     
    21     32        6   4   11  4294966307           0     
    22     32        6   4   11  4294966410           0     
    23     32        6   4   11  4294966332           0     
    24     32        6   4   11  4294966404           0     
    25     32        6   4   11  4294966324           0     
    26     32        6   4   11  4294966334           0     
    27     32        6   4   11  4294966353           0     
    28     32        6   4   11  4294966342           0     
    29     32        6   4   11  4294966336           0     
    30     32        6   4   11  4294966305           0     
    31     32        6   4   11  4294966296           0     
    32     32        6   4   11  4294966370           1      0:nan
    33     32        6   4   11  4294966376           0     
    34     32        6   4   11  4294966338           0     
    35     32        6   4   11  4294966342           0     
    36     32        6   4   11  4294966303           0     
    37     32        6   4   11  4294966320           0     
    38     32        6   4   11  4294966329           0     
    39     32        6   4   11  4294966299           0     
    40     32        6   4   11  4294966296           0     
    41     32        6   4   11  4294966305           0     
    42     32        6   4   11  4294966300           0     
    43     32        6   4   11  4294966384           0     
    44     32        6   4   11  4294966452           0     
    45     32        6   4   11  4294966310           0     
    46     32        6   4   11  4294966306           1      28:nan
    47     32        6   4   11  4294966346           0     
    48     32        6   4   11  4294966370           0     
    49     32        6   4   11  4294966320           0     
    50     32        6   4   11  4294966313           0     
    51     32        6   4   11  4294966350           0     
    52     32        6   4   11  4294966302           0     
    53     32        6   4   11  4294966377           0     
    54     32        6   4   11  4294966299           0     
    55     32        6   4   11  4294966322           0     
    56     32        6   4   11  4294966306           0     
    57     32        6   4   11  4294966356           0     
    58     32        6   4   11  4294966307           0     
    59     32        6   4   11  4294966342           0     
    60     32        6   4   11  4294966334           0     
    61     32        6   4   11  4294966311           0     
    62     32        6   4   11  4294966298           4      16:inf, 17:-inf, 18:-0, 19:1.4012984643248171e-45
    63     32        6   4   11  4294966306           1      31:nan
    64     32        6   4   11  4294966349           0     
    65     32        6   4   11  4294966298           0     
    66     32        6   4   11  4294966332           0     
    67     32        6   4   11  4294966302           0     
    68     32        6   4   11  4294966312           0     
    69     32        6   4   11  4294966332           0     
    70     32        6   4   11  4294966317           0     
    71     32        6   4   11  4294966316           0     
    72     32        6   4   11  4294966379           0     
    73     32        6   4   11  4294966501           0     
    74     32        6   4   11  4294966306           0     
    75     32        6   4   11  4294966297           0     
    76     32        6   4   11  4294966304           0     
    77     32        6   4   11  4294966302           0     
    78     32        6   4   11  4294966330           1      4:3.1415927410125732
    79     32        6   4   11  4294966447           0     
    80     32        6   4   11  4294966297           0     
    81     32        6   4   11  4294966309           0     
    82     32        6   4   11  4294966428           0     
    83     32        6   4   11  4294966414           0     
    84     32        6   4   11  4294966424           0     
    85     32        6   4   11  4294966381           0     
    86     32        6   4   11  4294966314           0     
    87     32        6   4   11  4294966397           0     
    88     32        6   4   11  4294966307           0     
    89     32        6   4   11  4294966316           0     
    90     32        6   4   11  4294966297           0     
    91     32        6   4   11  4294966305           0     
    92     32        6   4   11  4294966345           0     
    93     32        6   4   11  4294966558           0     
    94     32        6   4   11  4294966331           0     
    95     32        6   4   11  4294966353           0     
    96     32        6   4   11  4294966350           1      28:44974936064
    97     32        6   4   11  4294966409           0     
    98     32        6   4   11  4294966307           0     
    99     32        6   4   11  4294966390           0     
    100    32        6   4   11  4294966338           0     
    101    32        6   4   11  4294966378           0     
    102    32        6   4   11  4294966387           0     
    103    32        6   4   11  4294966360           0     
    104    32        6   4   11  4294966312           0     
    105    32        6   4   11  4294966338           0     
    106    32        6   4   11  4294966433           0     
    107    32        6   4   11  4294966414           0     
    108    32        6   4   11  4294966334           0     
    109    32        6   4   11  4294966498           0     
    110    32        6   4   11  4294966314           0     
    111    32        6   4   11  4294966407           0     
    112    32        6   4   11  4294966427           0     
    113    32        6   4   11  4294966333           0     
    114    32        6   4   11  4294966314           0     
    115    32        6   4   11  4294966345           1      31:-1243432448
    116    32        6   4   11  4294966483           0     
    117    32        6   4   11  4294966458           0     
    118    32        6   4   11  4294966316           0     
    119    32        6   4   11  4294966323           0     
    120    32        6   4   11  4294966317           0     
    121    32        6   4   11  4294966327           0     
    122    32        6   4   11  4294966303           0     
    123    32        6   4   11  4294966335           0     
    124    32        6   4   11  4294966555           0     
    125    32        6   4   11  4294966426           0     
    126    32        6   4   11  4294966336           0     
    127    32        6   4   11  4294966296           0     
    128    32        6   4   11  4294966437           16     0:-5.6894025802612305, 2:-5.7471122741699219, 4:5.2631058692932129, 6:3.063255786895752, 8:7.4240894317626953, ...
    129    32        6   4   11  4294966560           16     0:8.3615312576293945, 2:-7.4509210586547852, 4:-4.1556353569030762, 6:6.5207810401916504, 8:6.0981354713439941, ...
    130    32        6   4   11  4294966563           16     0:-4.0016179084777832, 2:1.9761660099029541, 4:8.0578994750976562, 6:7.5452351570129395, 8:1.8644313812255859, ...
    131    32        6   4   11  4294966474           16     0:-8.7062625885009766, 2:1.435447096824646, 4:3.8890092372894287, 6:-3.6610684394836426, 8:-0.71133530139923096, ...
    132    32        6   4   11  4294966388           16     0:-4.6208209991455078, 2:-8.5813560485839844, 4:-9.9415788650512695, 6:-0.55013293027877808, 8:8.1476030349731445, ...
    133    32        6   4   11  4294966677           16     0:0.25335460901260376, 2:-9.1122446060180664, 4:-2.0596992969512939, 6:-4.3087539672851562, 8:-7.7232322692871094, ...
    134    32        6   4   11  4294966715           16     0:4.4278688430786133, 2:2.7280275821685791, 4:-4.8478555679321289, 6:-9.1888256072998047, 8:3.7317984104156494, ...
    135    32        6   4   11  4294966368           16     0:-7.6852478981018066, 2:7.4739718437194824, 4:2.7575418949127197, 6:8.0490665435791016, 8:6.5803546905517578, ...
    136    32        6   4   11  4294966974           16     0:7.2348499298095703, 2:-1.0716096162796021, 4:-2.3152074813842773, 6:7.1512470245361328, 8:5.1870203018188477, ...
    137    32        6   4   11  4294966370           16     0:-1.8074220418930054, 2:-7.9748015403747559, 4:2.1155848503112793, 6:6.8344674110412598, 8:1.1937688589096069, ...
    138    32        6   4   11  4294966363           16     0:9.1183767318725586, 2:7.768073558807373, 4:9.8971595764160156, 6:-8.2359657287597656, 8:9.2227096557617188, ...
    139    32        6   4   11  4294966359           16     0:0.89655309915542603, 2:1.6093521118164062, 4:9.9942760467529297, 6:2.524686336517334, 8:-5.8538198471069336, ...
    140    32        6   4   11  4294966690           16     0:4.9172005653381348, 2:-7.5035834312438965, 4:3.6151180267333984, 6:7.541811466217041, 8:9.3921546936035156, ...
    141    32        6   4   11  4294966411           16     0:2.042339563369751, 2:6.1317868232727051, 4:-5.9905099868774414, 6:-8.8105688095092773, 8:-1.2170037031173706, ...
    142    32        6   4   11  4294966296           16     0:-4.4248027801513672, 2:0.57156217098236084, 4:-9.8227348327636719, 6:-5.9869794845581055, 8:6.5693860054016113, ...
    143    32        6   4   11  4294966340           16     0:-3.0235893726348877, 2:2.6417965888977051, 4:8.9537601470947266, 6:-7.9811758995056152, 8:-4.0086865425109863, ...
    144    32        6   4   11  4294966455           16     0:9.5796222686767578, 2:-5.7074384689331055, 4:2.9563488960266113, 6:-6.2431430816650391, 8:0.061678990721702576, ...
    145    32        6   4   11  4294966472           16     0:2.5911538600921631, 2:-4.569058895111084, 4:-3.6482639312744141, 6:-6.2192096710205078, 8:-9.0629615783691406, ...
    146    32        6   4   11  4294966470           16     0:-2.453639030456543, 2:3.5028011798858643, 4:3.7403855323791504, 6:1.2925810813903809, 8:6.8849740028381348, ...
    147    32        6   4   11  4294966505           16     0:-0.48064544796943665, 2:-5.8223457336425781, 4:5.4156122207641602, 6:-8.9528999328613281, 8:-6.2795400619506836, ...
    148    32        6   4   11  4294966312           16     0:-8.2931728363037109, 2:-2.6442804336547852, 4:6.4661884307861328, 6:0.31723803281784058, 8:-8.0026636123657227, ...
    149    32        6   4   11  4294966328           16     0:5.4584298133850098, 2:-7.3696346282958984, 4:-7.4889955520629883, 6:-8.4723014831542969, 8:6.1431107521057129, ...
    150    32        6   4   11  4294966528           16     0:-0.84008479118347168, 2:-6.9534440040588379, 4:-8.6861028671264648, 6:8.7475690841674805, 8:-0.31845855712890625, ...
    151    32        6   4   11  4294966300           16     0:-7.0607457160949707, 2:0.5453035831451416, 4:-2.2366857528686523, 6:2.8563783168792725, 8:-3.0961558818817139, ...
    152    32        6   4   11  4294966505           16     0:-4.5059237480163574, 2:5.2170867919921875, 4:0.93154287338256836, 6:-0.82804101705551147, 8:-6.6850042343139648, ...
    153    32        6   4   11  4294966381           16     0:-6.4780902862548828, 2:2.4468653202056885, 4:-6.1978468894958496, 6:4.6424522399902344, 8:8.8740320205688477, ...
    154    32        6   4   11  4294966424           16     0:8.5238809585571289, 2:6.095062255859375, 4:-5.5584826469421387, 6:-6.442237377166748, 8:4.5772285461425781, ...
    155    32        6   4   11  4294966569           16     0:2.440791130065918, 2:-4.9799723625183105, 4:1.4768251180648804, 6:3.1326212882995605, 8:8.0447282791137695, ...
    156    32        6   4   11  4294966335           16     0:-1.3979425430297852, 2:8.9291305541992188, 4:-1.981164813041687, 6:5.6073098182678223, 8:7.7777485847473145, ...
    157    32        6   4   11  4294966574           16     0:0.31318798661231995, 2:0.78189116716384888, 4:-4.1994161605834961, 6:-9.2485780715942383, 8:0.93401038646697998, ...
    158    32        6   4   11  4294966305           16     0:-3.9413819313049316, 2:6.36053466796875, 4:6.3355894088745117, 6:-7.4928703308105469, 8:8.0388040542602539, ...
    159    32        6   4   11  4294966404           16     0:4.2884831428527832, 2:1.201833963394165, 4:0.27708917856216431, 6:-6.5582156181335449, 8:-1.0237270593643188, ...
    160    32        6   4   0   0                    32     0:0.98619550466537476, 1:-3.7603194713592529, 2:-0.34999150037765503, 3:0.1334528774023056, 4:-2.2552545070648193, ...
    161    32        6   4   0   0                    32     0:0.030309226363897324, 1:-8.5821685791015625, 2:-3.74273681640625, 3:8.7059421539306641, 4:7.0136051177978516, ...
    162    32        6   4   0   0                    32     0:2.0220539569854736, 1:9.8072357177734375, 2:-3.4949219226837158, 3:8.8254489898681641, 4:8.2946224212646484, ...
    163    32        6   4   0   0                    32     0:1.0675803422927856, 1:2.8326382637023926, 2:4.0758461952209473, 3:-2.5348036289215088, 4:1.2731994390487671, ...
    164    32        6   4   0   0                    32     0:3.2998902797698975, 1:-0.039192099124193192, 2:4.8564996719360352, 3:-3.4591071605682373, 4:7.801140308380127, ...
    165    32        6   4   0   0                    32     0:-3.3393025398254395, 1:7.6992859840393066, 2:-2.334315299987793, 3:-0.12998028099536896, 4:-4.3339047431945801, ...
    166    32        6   4   0   0                    32     0:1.9769080877304077, 1:-5.9780254364013672, 2:1.8046531677246094, 3:-1.847342848777771, 4:-0.60948920249938965, ...
    167    32        6   4   0   0                    32     0:5.7271547317504883, 1:2.6443319320678711, 2:-2.5295965671539307, 3:-4.5700249671936035, 4:-5.3643283843994141, ...
    168    32        6   4   0   0                    32     0:4.6482772827148438, 1:7.6332497596740723, 2:-7.5700922012329102, 3:2.6927852630615234, 4:1.642876148223877, ...
    169    32        6   4   0   0                    32     0:0.24569518864154816, 1:-7.7142844200134277, 2:5.2960195541381836, 3:3.5779545307159424, 4:5.4597344398498535, ...
    170    32        6   4   0   0                    32     0:-4.3823604583740234, 1:-5.8408727645874023, 2:-9.1602916717529297, 3:-0.41479822993278503, 4:-6.3655343055725098, ...
    171    32        6   4   0   0                    32     0:-9.0268468856811523, 1:1.6203017234802246, 2:-1.4785804748535156, 3:0.9606471061706543, 4:0.42146375775337219, ...
    172    32        6   4   0   0                    32     0:4.4868879318237305, 1:4.9150676727294922, 2:1.1585489511489868, 3:1.693361759185791, 4:-7.3269882202148438, ...
    173    32        6   4   0   0                    32     0:-1.6951805353164673, 1:7.769282341003418, 2:6.4865379333496094, 3:2.6007976531982422, 4:7.1624078750610352, ...
    174    32        6   4   0   0                    32     0:-8.3955554962158203, 1:-3.7560725212097168, 2:-9.6851415634155273, 3:-9.6570043563842773, 4:8.3320655822753906, ...
    175    32        6   4   0   0                    32     0:0.080075323581695557, 1:8.984013557434082, 2:6.6566381454467773, 3:-2.7513735294342041, 4:7.2728285789489746, ...
    176    32        6   4   0   0                    32     0:1.6899958848953247, 1:6.6758389472961426, 2:2.8515520095825195, 3:-8.4288434982299805, 4:4.8112945556640625, ...
    177    32        6   4   0   0                    32     0:-2.080768346786499, 1:3.3861720561981201, 2:7.8378620147705078, 3:-3.8789706230163574, 4:1.0375427007675171, ...
    178    32        6   4   0   0                    32     0:9.7345819473266602, 1:-4.5726804733276367, 2:-8.0283412933349609, 3:6.5862846374511719, 4:4.1335105895996094, ...
    179    32        6   4   0   0                    32     0:0.4552028477191925, 1:-0.58265292644500732, 2:3.1211004257202148, 3:1.1915770769119263, 4:3.7407829761505127, ...
    180    32        6   4   0   0                    32     0:1.5322799682617188, 1:1.1753189563751221, 2:-2.5766096115112305, 3:5.7128620147705078, 4:-4.9440312385559082, ...
    181    32        6   4   0   0                    32     0:5.7162528038024902, 1:1.7989577054977417, 2:-9.8712959289550781, 3:5.4503259658813477, 4:4.049767017364502, ...
    182    32        6   4   0   0                    32     0:5.467620849609375, 1:8.2993173599243164, 2:-6.2172179222106934, 3:8.4473495483398438, 4:3.9124929904937744, ...
    183    32        6   4   0   0                    32     0:1.9925512075424194, 1:0.12660616636276245, 2:-2.9917349815368652, 3:8.2586202621459961, 4:-3.2045953273773193, ...
    184    32        6   4   0   0                    32     0:8.0953264236450195, 1:2.9503293037414551, 2:-4.5391874313354492, 3:-6.4561500549316406, 4:-4.1146655082702637, ...
    185    32        6   4   0   0                    32     0:1.6721007823944092, 1:-2.1975953578948975, 2:-2.8973672389984131, 3:-9.0950164794921875, 4:-5.6418256759643555, ...
    186    32        6   4   0   0                    32     0:1.2251135110855103, 1:-5.0751166343688965, 2:-1.5290416479110718, 3:5.8714132308959961, 4:-9.3644304275512695, ...
    187    32        6   4   0   0                    32     0:5.2376723289489746, 1:5.6301131248474121, 2:-4.4848518371582031, 3:-5.915064811706543, 4:-9.6648740768432617, ...
    188    32        6   4   0   0                    32     0:-1.1176950931549072, 1:5.0005683898925781, 2:0.633220374584198, 3:0.75565874576568604, 4:-4.4380097389221191, ...
    189    32        6   4   0   0                    32     0:3.1610996723175049, 1:-9.8388175964355469, 2:8.8634967803955078, 3:-5.257110595703125, 4:-7.8477163314819336, ...
    190    32        6   4   0   0                    32     0:-1.3803218603134155, 1:-3.9101493358612061, 2:-6.0380897521972656, 3:-8.6861581802368164, 4:7.7423863410949707, ...
    191    32        6   4   0   0                    32     0:-9.8062620162963867, 1:7.1591019630432129, 2:4.6231470108032227, 3:8.9351205825805664, 4:5.6323094367980957, ...
  Row group 1: rows=1024 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=2754
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=5 (vector_size=32) num_elements=1024
    num_vectors=32 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        6   2   18  4294871265           0     
    1      32        6   2   18  4294867583           0     
    2      32        6   2   18  4294867616           0     
    3      32        6   2   18  4294882325           0     
    4      32        6   2   18  4294872839           0     
    5      32        6   2   18  4294873934           0     
    6      32        6   2   18  4294877022           0     
    7      32        6   2   18  4294871338           0     
    8      32        6   2   18  4294869331           0     
    9      32        6   2   18  4294868762           0     
    10     32        6   2   18  4294878646           0     
    11     32        6   2   18  4294867686           0     
    12     32        6   2   18  4294874507           0     
    13     32        6   2   18  4294868480           0     
    14     32        6   2   18  4294868220           0     
    15     32        6   2   18  4294873766           0     
    16     32        6   2   18  4294874531           0     
    17     32        6   2   18  4294877239           0     
    18     32        6   2   18  4294873026           0     
    19     32        6   2   18  4294868369           0     
    20     32        6   2   18  4294868197           0     
    21     32        6   2   18  4294867811           0     
    22     32        6   2   18  4294867362           0     
    23     32        6   2   18  4294868027           0     
    24     32        6   2   18  4294868758           0     
    25     32        6   2   18  4294867759           0     
    26     32        6   2   18  4294876098           0     
    27     32        6   2   18  4294871071           0     
    28     32        6   2   18  4294875247           0     
    29     32        6   2   18  4294867511           0     
    30     32        6   2   18  4294883872           0     
    31     32        6   2   18  4294868597           0     
  Row group 2: rows=1024 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=450
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=5 (vector_size=32) num_elements=1024
    num_vectors=32 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        10  8   0   777                  0     
    1      32        10  8   0   777                  0     
    2      32        10  8   0   777                  0     
    3      32        10  8   0   777                  0     
    4      32        10  8   0   777                  0     
    5      32        10  8   0   777                  0     
    6      32        10  8   0   777                  0     
    7      32        10  8   0   777                  0     
    8      32        10  8   0   777                  0     
    9      32        10  8   0   777                  0     
    10     32        10  8   0   777                  0     
    11     32        10  8   0   777                  0     
    12     32        10  8   0   777                  0     
    13     32        10  8   0   777                  0     
    14     32        10  8   0   777                  0     
    15     32        10  8   0   777                  0     
    16     32        10  8   0   777                  0     
    17     32        10  8   0   777                  0     
    18     32        10  8   0   777                  0     
    19     32        10  8   0   777                  0     
    20     32        10  8   0   777                  0     
    21     32        10  8   0   777                  0     
    22     32        10  8   0   777                  0     
    23     32        10  8   0   777                  0     
    24     32        10  8   0   777                  0     
    25     32        10  8   0   777                  0     
    26     32        10  8   0   777                  0     
    27     32        10  8   0   777                  0     
    28     32        10  8   0   777                  0     
    29     32        10  8   0   777                  0     
    30     32        10  8   0   777                  0     
    31     32        10  8   0   777                  0     
  Row group 3: rows=808 encodings=[ RLE ALP ] compression=uncompressed null_count=8 compressed_bytes=1498
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=5 (vector_size=32) num_elements=800
    num_vectors=25 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        6   4   11  4294966356           0     
    1      32        6   4   11  4294966297           0     
    2      32        6   4   11  4294966322           0     
    3      32        6   4   11  4294966327           0     
    4      32        6   4   11  4294966308           0     
    5      32        6   4   11  4294966336           0     
    6      32        6   4   11  4294966368           0     
    7      32        6   4   11  4294966387           0     
    8      32        6   4   11  4294966439           0     
    9      32        6   4   11  4294966363           0     
    10     32        6   4   11  4294966373           0     
    11     32        6   4   11  4294966379           0     
    12     32        6   4   11  4294966388           0     
    13     32        6   4   11  4294966314           0     
    14     32        6   4   11  4294966507           0     
    15     32        6   4   11  4294966352           0     
    16     32        6   4   11  4294966320           0     
    17     32        6   4   11  4294966383           0     
    18     32        6   4   11  4294966300           0     
    19     32        6   4   11  4294966311           0     
    20     32        6   4   11  4294966306           0     
    21     32        6   4   11  4294966363           0     
    22     32        6   4   11  4294966366           0     
    23     32        6   4   11  4294966299           0     
    24     32        6   4   11  4294966400           0     
  Row group 4: rows=32 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=237
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=5 (vector_size=32) num_elements=32
    num_vectors=1 total_exceptions=32
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        10  10  0   0                    32     0:-7.9999998744539955e+18, 1:7.9999998744539955e+18, 2:3.2175613495004365e+18, 3:-1.428295291294974e+18, 4:74196599420485632, ...

Column 'double_alp_32' (DOUBLE)
  Row group 0: rows=6144 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=25800
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=5 (vector_size=32) num_elements=6144
    num_vectors=192 total_exceptions=1546
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        14  12  11  18446744073709550703 0     
    1      32        14  12  11  18446744073709550636 0     
    2      32        14  12  11  18446744073709550629 0     
    3      32        14  12  11  18446744073709550630 0     
    4      32        14  12  11  18446744073709550656 0     
    5      32        14  12  11  18446744073709550822 0     
    6      32        14  12  11  18446744073709550711 0     
    7      32        14  12  11  18446744073709550805 0     
    8      32        14  12  11  18446744073709550617 0     
    9      32        14  12  11  18446744073709550738 0     
    10     32        14  12  11  18446744073709550759 0     
    11     32        14  12  11  18446744073709550617 0     
    12     32        14  12  11  18446744073709550768 0     
    13     32        14  12  11  18446744073709550632 0     
    14     32        14  12  11  18446744073709550633 0     
    15     32        14  12  11  18446744073709550770 0     
    16     32        14  12  11  18446744073709550635 0     
    17     32        14  12  11  18446744073709550620 0     
    18     32        14  12  11  18446744073709550732 0     
    19     32        14  12  11  18446744073709550709 0     
    20     32        14  12  11  18446744073709550638 0     
    21     32        14  12  11  18446744073709550627 0     
    22     32        14  12  11  18446744073709550730 0     
    23     32        14  12  11  18446744073709550652 0     
    24     32        14  12  11  18446744073709550724 0     
    25     32        14  12  11  18446744073709550644 0     
    26     32        14  12  11  18446744073709550654 0     
    27     32        14  12  11  18446744073709550673 0     
    28     32        14  12  11  18446744073709550662 0     
    29     32        14  12  11  18446744073709550656 0     
    30     32        14  12  11  18446744073709550625 0     
    31     32        14  12  11  18446744073709550616 0     
    32     32        14  12  11  18446744073709550690 1      0:nan
    33     32        14  12  11  18446744073709550696 0     
    34     32        14  12  11  18446744073709550658 0     
    35     32        14  12  11  18446744073709550662 0     
    36     32        14  12  11  18446744073709550623 0     
    37     32        14  12  11  18446744073709550640 0     
    38     32        14  12  11  18446744073709550649 0     
    39     32        14  12  11  18446744073709550619 0     
    40     32        14  12  11  18446744073709550616 0     
    41     32        14  12  11  18446744073709550625 0     
    42     32        14  12  11  18446744073709550620 0     
    43     32        14  12  11  18446744073709550704 0     
    44     32        14  12  11  18446744073709550772 0     
    45     32        14  12  11  18446744073709550630 0     
    46     32        14  12  11  18446744073709550626 1      28:nan
    47     32        14  12  11  18446744073709550666 0     
    48     32        14  12  11  18446744073709550690 0     
    49     32        14  12  11  18446744073709550640 0     
    50     32        14  12  11  18446744073709550633 0     
    51     32        14  12  11  18446744073709550670 0     
    52     32        14  12  11  18446744073709550622 0     
    53     32        14  12  11  18446744073709550697 0     
    54     32        14  12  11  18446744073709550619 0     
    55     32        14  12  11  18446744073709550642 0     
    56     32        14  12  11  18446744073709550626 0     
    57     32        14  12  11  18446744073709550676 0     
    58     32        14  12  11  18446744073709550627 0     
    59     32        14  12  11  18446744073709550662 0     
    60     32        14  12  11  18446744073709550654 0     
    61     32        14  12  11  18446744073709550631 0     
    62     32        14  12  11  18446744073709550618 4      16:inf, 17:-inf, 18:-0, 19:4.9406564584124654e-324
    63     32        14  12  11  18446744073709550626 1      31:nan
    64     32        14  12  11  18446744073709550669 0     
    65     32        14  12  11  18446744073709550618 0     
    66     32        14  12  11  18446744073709550652 0     
    67     32        14  12  11  18446744073709550622 0     
    68     32        14  12  11  18446744073709550632 0     
    69     32        14  12  11  18446744073709550652 0     
    70     32        14  12  11  18446744073709550637 0     
    71     32        14  12  11  18446744073709550636 0     
    72     32        14  12  11  18446744073709550699 0     
    73     32        14  12  11  18446744073709550821 0     
    74     32        14  12  11  18446744073709550626 0     
    75     32        14  12  11  18446744073709550617 0     
    76     32        14  12  11  18446744073709550624 0     
    77     32        14  12  11  18446744073709550622 0     
    78     32        14  12  11  18446744073709550650 1      4:3.1415926535897931
    79     32        14  12  11  18446744073709550767 0     
    80     32        14  12  11  18446744073709550617 0     
    81     32        14  12  11  18446744073709550629 0     
    82     32        14  12  11  18446744073709550748 0     
    83     32        14  12  11  18446744073709550734 0     
    84     32        14  12  11  18446744073709550744 0     
    85     32        14  12  11  18446744073709550701 0     
    86     32        14  12  11  18446744073709550634 0     
    87     32        14  12  11  18446744073709550717 0     
    88     32        14  12  11  18446744073709550627 0     
    89     32        14  12  11  18446744073709550636 0     
    90     32        14  12  11  18446744073709550617 0     
    91     32        14  12  11  18446744073709550625 0     
    92     32        14  12  11  18446744073709550665 0     
    93     32        14  12  11  18446744073709550878 0     
    94     32        14  12  11  18446744073709550651 0     
    95     32        14  12  11  18446744073709550673 0     
    96     32        14  12  11  18446744073709550670 1      28:44974934523.343002
    97     32        14  12  11  18446744073709550729 0     
    98     32        14  12  11  18446744073709550627 0     
    99     32        14  12  11  18446744073709550710 0     
    100    32        14  12  11  18446744073709550658 0     
    101    32        14  12  11  18446744073709550698 0     
    102    32        14  12  11  18446744073709550707 0     
    103    32        14  12  11  18446744073709550680 0     
    104    32        14  12  11  18446744073709550632 0     
    105    32        14  12  11  18446744073709550658 0     
    106    32        14  12  11  18446744073709550753 0     
    107    32        14  12  11  18446744073709550734 0     
    108    32        14  12  11  18446744073709550654 0     
    109    32        14  12  11  18446744073709550818 0     
    110    32        14  12  11  18446744073709550634 0     
    111    32        14  12  11  18446744073709550727 0     
    112    32        14  12  11  18446744073709550747 0     
    113    32        14  12  11  18446744073709550653 0     
    114    32        14  12  11  18446744073709550634 0     
    115    32        14  12  11  18446744073709550665 1      31:-1243432432.3432
    116    32        14  12  11  18446744073709550803 0     
    117    32        14  12  11  18446744073709550778 0     
    118    32        14  12  11  18446744073709550636 0     
    119    32        14  12  11  18446744073709550643 0     
    120    32        14  12  11  18446744073709550637 0     
    121    32        14  12  11  18446744073709550647 0     
    122    32        14  12  11  18446744073709550623 0     
    123    32        14  12  11  18446744073709550655 0     
    124    32        14  12  11  18446744073709550875 0     
    125    32        14  12  11  18446744073709550746 0     
    126    32        14  12  11  18446744073709550656 0     
    127    32        14  12  11  18446744073709550616 0     
    128    32        14  12  11  18446744073709550757 16     0:-5.6894026020883821, 2:-5.7471123414096521, 4:5.2631058917995048, 6:3.0632557599260757, 8:7.4240896660071236, ...
    129    32        14  12  11  18446744073709550880 16     0:8.3615310165433918, 2:-7.4509208403525999, 4:-4.1556354486025482, 6:6.5207809345366901, 8:6.0981354522486946, ...
    130    32        14  12  11  18446744073709550883 16     0:-4.0016177565494946, 2:1.9761659612096527, 4:8.0578997873910296, 6:7.5452351656382159, 8:1.8644314134978135, ...
    131    32        14  12  11  18446744073709550794 16     0:-8.7062622521571491, 2:1.4354470416709053, 4:3.8890091633305257, 6:-3.6610683282734504, 8:-0.71133532824648515, ...
    132    32        14  12  11  18446744073709550708 16     0:-4.6208211430056672, 2:-8.5813557849872311, 4:-9.941579321146051, 6:-0.55013293847550604, 8:8.1476033946762261, ...
    133    32        14  12  11  18446744073709550997 16     0:0.2533545960602801, 2:-9.1122445962695, 4:-2.0596992584046814, 6:-4.3087539817807592, 8:-7.7232320987702971, ...
    134    32        14  12  11  18446744073709551035 16     0:4.4278690443430024, 2:2.7280275284446809, 4:-4.8478556577229304, 6:-9.1888252642959145, 8:3.7317983148228273, ...
    135    32        14  12  11  18446744073709550688 16     0:-7.6852479162007246, 2:7.4739719701666463, 4:2.7575418619624026, 6:8.0490660748490193, 8:6.5803545809650981, ...
    136    32        14  12  11  18446744073709551294 16     0:7.2348498055074062, 2:-1.0716095772092904, 4:-2.315207571730109, 6:7.1512467947328613, 8:5.1870204923517775, ...
    137    32        14  12  11  18446744073709550690 16     0:-1.8074220521761097, 2:-7.974801732640632, 4:2.1155849324871756, 6:6.834467273971967, 8:1.1937688569912508, ...
    138    32        14  12  11  18446744073709550683 16     0:9.118376566890614, 2:7.7680734846461874, 4:9.8971594199675277, 6:-8.2359658322110167, 8:9.2227097535826132, ...
    139    32        14  12  11  18446744073709550679 16     0:0.89655311572229834, 2:1.6093520595004884, 4:9.9942761479839533, 6:2.5246863726085111, 8:-5.8538199798720996, ...
    140    32        14  12  11  18446744073709551010 16     0:4.91720075314237, 2:-7.5035836009805905, 4:3.6151179289175683, 6:7.5418116779382434, 8:9.3921545424686848, ...
    141    32        14  12  11  18446744073709550731 16     0:2.0423396622180889, 2:6.1317867346737938, 4:-5.9905097657978379, 6:-8.8105687167995406, 8:-1.2170036882014579, ...
    142    32        14  12  11  18446744073709550616 16     0:-4.4248025522874324, 2:0.57156218779601531, 4:-9.8227343764463715, 6:-5.9869796689701227, 8:6.5693860547367873, ...
    143    32        14  12  11  18446744073709550660 16     0:-3.0235893916984216, 2:2.6417965307412583, 4:8.9537604738820384, 6:-7.9811759900529164, 8:-4.0086864686154566, ...
    144    32        14  12  11  18446744073709550775 16     0:9.5796223926324124, 2:-5.7074385015131295, 4:2.9563489203160698, 6:-6.2431429413942592, 8:0.061678992210847383, ...
    145    32        14  12  11  18446744073709550792 16     0:2.5911538387824695, 2:-4.5690591258260795, 4:-3.6482640430217295, 6:-6.2192095071071076, 8:-9.0629618159354823, ...
    146    32        14  12  11  18446744073709550790 16     0:-2.4536389886526688, 2:3.5028012979965584, 4:3.7403854407184878, 6:1.2925810948657523, 8:6.8849740579384928, ...
    147    32        14  12  11  18446744073709550825 16     0:-0.48064543664115833, 2:-5.8223455347540902, 4:5.4156123212012801, 6:-8.952899598503107, 8:-6.2795400024970638, ...
    148    32        14  12  11  18446744073709550632 16     0:-8.2931732158632254, 2:-2.6442803347343258, 4:6.4661886545740632, 6:0.31723801876425428, 8:-8.0026636353340344, ...
    149    32        14  12  11  18446744073709550648 16     0:5.458429671205824, 2:-7.3696346789538119, 4:-7.4889956654183116, 6:-8.4723011729449613, 8:6.1431108293243692, ...
    150    32        14  12  11  18446744073709550848 16     0:-0.84008481683608083, 2:-6.9534442089590929, 4:-8.6861029132294689, 6:8.7475688579875754, 8:-0.3184585478003521, ...
    151    32        14  12  11  18446744073709550620 16     0:-7.0607459066610909, 2:0.54530355991387669, 4:-2.2366858324668328, 6:2.8563783069799209, 8:-3.0961558930116695, ...
    152    32        14  12  11  18446744073709550825 16     0:-4.5059238112489295, 2:5.217086960594008, 4:0.93154285823395355, 6:-0.82804099455095814, 8:-6.6850043946836397, ...
    153    32        14  12  11  18446744073709550701 16     0:-6.4780904790686398, 2:2.4468653891885506, 4:-6.1978471138785123, 6:4.6424521356476234, 8:8.8740317586578481, ...
    154    32        14  12  11  18446744073709550744 16     0:8.52388143249315, 2:6.0950624860295877, 4:-5.5584825992302189, 6:-6.4422376112934669, 8:4.5772287599494854, ...
    155    32        14  12  11  18446744073709550889 16     0:2.4407912062346981, 2:-4.9799722992235687, 4:1.4768251244168451, 6:3.1326213638093514, 8:8.0447280418146345, ...
    156    32        14  12  11  18446744073709550655 16     0:-1.3979425730507333, 2:8.9291301691148313, 4:-1.9811647902589369, 6:5.6073099550004502, 8:7.7777484966271349, ...
    157    32        14  12  11  18446744073709550894 16     0:0.31318799197485259, 2:0.78189115255742125, 4:-4.1994161005948092, 6:-9.2485782274463677, 8:0.93401039078769621, ...
    158    32        14  12  11  18446744073709550625 16     0:-3.9413819387106641, 2:6.3605348967262305, 4:6.3355894119144125, 6:-7.4928704268223889, 8:8.0388039086804, ...
    159    32        14  12  11  18446744073709550724 16     0:4.2884833767645052, 2:1.2018339557526869, 4:0.27708918219772949, 6:-6.5582153829331951, 8:-1.0237270643332752, ...
    160    32        14  12  0   0                    32     0:0.98619550441899628, 1:-3.7603195733564965, 2:-0.34999150139969903, 3:0.13345287723017574, 4:-2.2552545378626987, ...
    161    32        14  12  0   0                    32     0:0.030309225771123494, 1:-8.5821684831907916, 2:-3.7427368143085946, 3:8.7059419910650426, 4:7.0136051698203472, ...
    162    32        14  12  0   0                    32     0:2.0220540312628255, 1:9.8072356875118452, 2:-3.4949218142877667, 3:8.8254490414462943, 4:8.2946221923083936, ...
    163    32        14  12  0   0                    32     0:1.0675802903645892, 1:2.8326382686728713, 2:4.0758461738708913, 3:-2.5348037203540308, 4:1.2731994420797244, ...
    164    32        14  12  0   0                    32     0:3.2998903374048272, 1:-0.039192097718393493, 2:4.8564995546281331, 3:-3.4591070495844689, 4:7.8011404866940603, ...
    165    32        14  12  0   0                    32     0:-3.3393025819608448, 1:7.6992861267109021, 2:-2.3343151863427347, 3:-0.12998027809437485, 4:-4.3339046885598895, ...
    166    32        14  12  0   0                    32     0:1.9769081168059244, 1:-5.9780254400246156, 2:1.8046532073106891, 3:-1.8473428184975349, 4:-0.60948918547399256, ...
    167    32        14  12  0   0                    32     0:5.7271546356045411, 1:2.6443319643230301, 2:-2.5295964611953825, 3:-4.5700249378307065, 4:-5.3643282505631404, ...
    168    32        14  12  0   0                    32     0:4.6482773086323785, 1:7.6332496986298608, 2:-7.5700921134951669, 3:2.6927851808921965, 4:1.6428762027311383, ...
    169    32        14  12  0   0                    32     0:0.24569519186426803, 1:-7.7142846015922162, 2:5.296019433959664, 3:3.5779544845953337, 4:5.459734467809632, ...
    170    32        14  12  0   0                    32     0:-4.3823605780407142, 1:-5.8408728185879406, 2:-9.1602915196354804, 3:-0.4147982323665611, 4:-6.3655341491596547, ...
    171    32        14  12  0   0                    32     0:-9.0268466642665715, 1:1.6203017675618803, 2:-1.4785804936358937, 3:0.96064713589713691, 4:0.42146377163920556, ...
    172    32        14  12  0   0                    32     0:4.486888097169988, 1:4.9150676024475111, 2:1.1585489622686085, 3:1.6933618163548947, 4:-7.3269883464269521, ...
    173    32        14  12  0   0                    32     0:-1.6951805537861009, 1:7.76928231063851, 2:6.4865378208616598, 3:2.6007976921151266, 4:7.1624078463715479, ...
    174    32        14  12  0   0                    32     0:-8.39555504120524, 1:-3.7560725740420544, 2:-9.685141643360172, 3:-9.657004335828212, 4:8.3320660505646522, ...
    175    32        14  12  0   0                    32     0:0.080075327275126451, 1:8.9840139404560819, 2:6.6566380693333427, 3:-2.75137352442934, 4:7.2728284712765916, ...
    176    32        14  12  0   0                    32     0:1.6899958306629714, 1:6.6758390542706803, 2:2.8515520751337164, 3:-8.428843366348616, 4:4.8112944261319512, ...
    177    32        14  12  0   0                    32     0:-2.0807682910219616, 1:3.3861721345082563, 2:7.8378619712430408, 3:-3.8789706074727457, 4:1.0375427318974562, ...
    178    32        14  12  0   0                    32     0:9.734582118059782, 1:-4.5726806337250148, 2:-8.0283414335793957, 3:6.5862847794058643, 4:4.1335106375739894, ...
    179    32        14  12  0   0                    32     0:0.45520285235329805, 1:-0.58265292789138612, 2:3.1211004068220105, 3:1.1915770708237261, 4:3.7407829122765968, ...
    180    32        14  12  0   0                    32     0:1.5322800256643609, 1:1.1753189143255649, 2:-2.5766095632162855, 3:5.7128620387194733, 4:-4.9440310924567834, ...
    181    32        14  12  0   0                    32     0:5.7162528403017054, 1:1.7989576463980339, 2:-9.871295818742043, 3:5.4503257361534025, 4:4.0497669426751131, ...
    182    32        14  12  0   0                    32     0:5.4676209982698705, 1:8.2993173231910582, 2:-6.2172180059909845, 3:8.4473497899298078, 4:3.9124930425244209, ...
    183    32        14  12  0   0                    32     0:1.9925512603817586, 1:0.12660616955186832, 2:-2.9917349399110904, 3:8.2586201293275927, 4:-3.2045952822937815, ...
    184    32        14  12  0   0                    32     0:8.0953267620034204, 1:2.9503292765057521, 2:-4.5391876634495887, 3:-6.4561500280262383, 4:-4.1146652777476245, ...
    185    32        14  12  0   0                    32     0:1.6721008306574325, 1:-2.1975952707390158, 2:-2.8973671535649537, 3:-9.0950162201076132, 4:-5.6418258913608375, ...
    186    32        14  12  0   0                    32     0:1.2251135674135649, 1:-5.0751167678352642, 2:-1.5290416971839027, 3:5.8714131917039269, 4:-9.3644304282047273, ...
    187    32        14  12  0   0                    32     0:5.2376723141578196, 1:5.6301129440991726, 2:-4.4848518145311438, 3:-5.9150648522037743, 4:-9.6648737977605546, ...
    188    32        14  12  0   0                    32     0:-1.1176951131509882, 1:5.0005681968987803, 2:0.63322039979124778, 3:0.75565874425611801, 4:-4.4380096874285506, ...
    189    32        14  12  0   0                    32     0:3.1610996562355953, 1:-9.8388178747476562, 2:8.86349644297872, 3:-5.2571106103962943, 4:-7.8477163080531094, ...
    190    32        14  12  0   0                    32     0:-1.3803218971563247, 1:-3.9101493632402851, 2:-6.038089650969348, 3:-8.6861577202510869, 4:7.7423864445777308, ...
    191    32        14  12  0   0                    32     0:-9.806262317234884, 1:7.1591018251743872, 2:4.6231469233923228, 3:8.9351207743550294, 4:5.6323095784117587, ...
  Row group 1: rows=1024 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=2882
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=5 (vector_size=32) num_elements=1024
    num_vectors=32 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        16  12  18  18446744073709455585 0     
    1      32        16  12  18  18446744073709451903 0     
    2      32        16  12  18  18446744073709451936 0     
    3      32        16  12  18  18446744073709466645 0     
    4      32        16  12  18  18446744073709457159 0     
    5      32        16  12  18  18446744073709458254 0     
    6      32        16  12  18  18446744073709461342 0     
    7      32        16  12  18  18446744073709455658 0     
    8      32        16  12  18  18446744073709453651 0     
    9      32        16  12  18  18446744073709453082 0     
    10     32        16  12  18  18446744073709462966 0     
    11     32        16  12  18  18446744073709452006 0     
    12     32        16  12  18  18446744073709458827 0     
    13     32        16  12  18  18446744073709452800 0     
    14     32        16  12  18  18446744073709452540 0     
    15     32        16  12  18  18446744073709458086 0     
    16     32        16  12  18  18446744073709458851 0     
    17     32        16  12  18  18446744073709461559 0     
    18     32        16  12  18  18446744073709457346 0     
    19     32        16  12  18  18446744073709452689 0     
    20     32        16  12  18  18446744073709452517 0     
    21     32        16  12  18  18446744073709452131 0     
    22     32        16  12  18  18446744073709451682 0     
    23     32        16  12  18  18446744073709452347 0     
    24     32        16  12  18  18446744073709453078 0     
    25     32        16  12  18  18446744073709452079 0     
    26     32        16  12  18  18446744073709460418 0     
    27     32        16  12  18  18446744073709455391 0     
    28     32        16  12  18  18446744073709459567 0     
    29     32        16  12  18  18446744073709451831 0     
    30     32        16  12  18  18446744073709468192 0     
    31     32        16  12  18  18446744073709452917 0     
  Row group 2: rows=1024 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=578
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=5 (vector_size=32) num_elements=1024
    num_vectors=32 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        16  14  0   777                  0     
    1      32        16  14  0   777                  0     
    2      32        16  14  0   777                  0     
    3      32        16  14  0   777                  0     
    4      32        16  14  0   777                  0     
    5      32        16  14  0   777                  0     
    6      32        16  14  0   777                  0     
    7      32        16  14  0   777                  0     
    8      32        16  14  0   777                  0     
    9      32        16  14  0   777                  0     
    10     32        16  14  0   777                  0     
    11     32        16  14  0   777                  0     
    12     32        16  14  0   777                  0     
    13     32        16  14  0   777                  0     
    14     32        16  14  0   777                  0     
    15     32        16  14  0   777                  0     
    16     32        16  14  0   777                  0     
    17     32        16  14  0   777                  0     
    18     32        16  14  0   777                  0     
    19     32        16  14  0   777                  0     
    20     32        16  14  0   777                  0     
    21     32        16  14  0   777                  0     
    22     32        16  14  0   777                  0     
    23     32        16  14  0   777                  0     
    24     32        16  14  0   777                  0     
    25     32        16  14  0   777                  0     
    26     32        16  14  0   777                  0     
    27     32        16  14  0   777                  0     
    28     32        16  14  0   777                  0     
    29     32        16  14  0   777                  0     
    30     32        16  14  0   777                  0     
    31     32        16  14  0   777                  0     
  Row group 3: rows=808 encodings=[ RLE ALP ] compression=uncompressed null_count=8 compressed_bytes=1598
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=5 (vector_size=32) num_elements=800
    num_vectors=25 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        14  12  11  18446744073709550676 0     
    1      32        14  12  11  18446744073709550617 0     
    2      32        14  12  11  18446744073709550642 0     
    3      32        14  12  11  18446744073709550647 0     
    4      32        14  12  11  18446744073709550628 0     
    5      32        14  12  11  18446744073709550656 0     
    6      32        14  12  11  18446744073709550688 0     
    7      32        14  12  11  18446744073709550707 0     
    8      32        14  12  11  18446744073709550759 0     
    9      32        14  12  11  18446744073709550683 0     
    10     32        14  12  11  18446744073709550693 0     
    11     32        14  12  11  18446744073709550699 0     
    12     32        14  12  11  18446744073709550708 0     
    13     32        14  12  11  18446744073709550634 0     
    14     32        14  12  11  18446744073709550827 0     
    15     32        14  12  11  18446744073709550672 0     
    16     32        14  12  11  18446744073709550640 0     
    17     32        14  12  11  18446744073709550703 0     
    18     32        14  12  11  18446744073709550620 0     
    19     32        14  12  11  18446744073709550631 0     
    20     32        14  12  11  18446744073709550626 0     
    21     32        14  12  11  18446744073709550683 0     
    22     32        14  12  11  18446744073709550686 0     
    23     32        14  12  11  18446744073709550619 0     
    24     32        14  12  11  18446744073709550720 0     
  Row group 4: rows=32 encodings=[ RLE ALP ] compression=uncompressed null_count=0 compressed_bytes=305
    ALP header: compression_mode=0 integer_encoding=0 log_vector_size=5 (vector_size=32) num_elements=32
    num_vectors=1 total_exceptions=0
    vector elems     exp fac bw  frame_of_ref         n_exc  first exceptions (pos:value)
    0      32        0   0   64  10446744073709551616 0     

Round-trip verification (bit-exact, including NaN/-0.0):
  float_plain      OK
  double_plain     OK
  float_alp_1024   OK
  double_alp_1024  OK
  float_alp_4096   OK
  double_alp_4096  OK
  float_alp_32     OK
  double_alp_32    OK

Comment thread data/README.md Outdated
Comment thread data/README.md
The same values appear in `PLAIN`-encoded reference columns so decoders can
bit-compare the `ALP` columns against known results:

| Column | Encoding | Rationale / coverage |

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to explain both what is in the file, as well as to explain the rationale for including each piece

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can verify this a bit with standard tools. For example, using the tools from the Rust implementation in apache/arrow-rs#9372

andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ ~/Software/arrow-rs/target/release/parquet-schema data/alp_extended.zstd.parquet
Metadata for file: data/alp_extended.zstd.parquet

version: 2
num of rows: 9000
created by: parquet-cpp-arrow version 23.0.0-SNAPSHOT
message schema {
  OPTIONAL FLOAT float_plain;
  OPTIONAL DOUBLE double_plain;
  OPTIONAL FLOAT float_alp_1024;
  OPTIONAL DOUBLE double_alp_1024;
  OPTIONAL FLOAT float_alp_4096;
  OPTIONAL DOUBLE double_alp_4096;
  OPTIONAL FLOAT float_alp_32;
  OPTIONAL DOUBLE double_alp_32;
}

@alamb alamb Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The layout:

andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ ~/Software/arrow-rs/target/release/parquet-layout data/alp_extended.zstd.parquet
Details
{
  "row_groups": [
    {
      "columns": [
        {
          "path": "float_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211391,
            "length": 12
          },
          "column_index": {
            "offset": 210367,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "double_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211403,
            "length": 14
          },
          "column_index": {
            "offset": 210395,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "float_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211417,
            "length": 14
          },
          "column_index": {
            "offset": 210431,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211431,
            "length": 14
          },
          "column_index": {
            "offset": 210459,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211445,
            "length": 14
          },
          "column_index": {
            "offset": 210495,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211459,
            "length": 14
          },
          "column_index": {
            "offset": 210523,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211473,
            "length": 14
          },
          "column_index": {
            "offset": 210559,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211487,
            "length": 14
          },
          "column_index": {
            "offset": 210587,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        }
      ],
      "row_count": 6144
    },
    {
      "columns": [
        {
          "path": "float_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211501,
            "length": 13
          },
          "column_index": {
            "offset": 210623,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "double_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211514,
            "length": 13
          },
          "column_index": {
            "offset": 210651,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "float_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211527,
            "length": 13
          },
          "column_index": {
            "offset": 210687,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211540,
            "length": 13
          },
          "column_index": {
            "offset": 210715,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211553,
            "length": 13
          },
          "column_index": {
            "offset": 210751,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211566,
            "length": 13
          },
          "column_index": {
            "offset": 210779,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211579,
            "length": 13
          },
          "column_index": {
            "offset": 210815,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211592,
            "length": 13
          },
          "column_index": {
            "offset": 210843,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        }
      ],
      "row_count": 1024
    },
    {
      "columns": [
        {
          "path": "float_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211605,
            "length": 12
          },
          "column_index": {
            "offset": 210879,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "double_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211617,
            "length": 12
          },
          "column_index": {
            "offset": 210907,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "float_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211629,
            "length": 12
          },
          "column_index": {
            "offset": 210943,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211641,
            "length": 12
          },
          "column_index": {
            "offset": 210971,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211653,
            "length": 12
          },
          "column_index": {
            "offset": 211007,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211665,
            "length": 12
          },
          "column_index": {
            "offset": 211035,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211677,
            "length": 13
          },
          "column_index": {
            "offset": 211071,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211690,
            "length": 13
          },
          "column_index": {
            "offset": 211099,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        }
      ],
      "row_count": 1024
    },
    {
      "columns": [
        {
          "path": "float_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211703,
            "length": 13
          },
          "column_index": {
            "offset": 211135,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "double_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211716,
            "length": 13
          },
          "column_index": {
            "offset": 211163,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "float_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211729,
            "length": 13
          },
          "column_index": {
            "offset": 211199,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211742,
            "length": 13
          },
          "column_index": {
            "offset": 211227,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211755,
            "length": 13
          },
          "column_index": {
            "offset": 211263,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211768,
            "length": 13
          },
          "column_index": {
            "offset": 211291,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211781,
            "length": 13
          },
          "column_index": {
            "offset": 211327,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 211794,
            "length": 13
          },
          "column_index": {
            "offset": 211355,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        }
      ],
      "row_count": 808
    }
  ],
  "footer": {
    "metadata_size": 3866
  }
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated layout

Details

{
  "row_groups": [
    {
      "columns": [
        {
          "path": "float_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213725,
            "length": 12
          },
          "column_index": {
            "offset": 212453,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "double_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213737,
            "length": 14
          },
          "column_index": {
            "offset": 212481,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "float_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213751,
            "length": 14
          },
          "column_index": {
            "offset": 212517,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213765,
            "length": 14
          },
          "column_index": {
            "offset": 212545,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213779,
            "length": 14
          },
          "column_index": {
            "offset": 212581,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213793,
            "length": 14
          },
          "column_index": {
            "offset": 212609,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213807,
            "length": 14
          },
          "column_index": {
            "offset": 212645,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213821,
            "length": 14
          },
          "column_index": {
            "offset": 212673,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        }
      ],
      "row_count": 6144
    },
    {
      "columns": [
        {
          "path": "float_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213835,
            "length": 13
          },
          "column_index": {
            "offset": 212709,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "double_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213848,
            "length": 13
          },
          "column_index": {
            "offset": 212737,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "float_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213861,
            "length": 13
          },
          "column_index": {
            "offset": 212773,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213874,
            "length": 13
          },
          "column_index": {
            "offset": 212801,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213887,
            "length": 13
          },
          "column_index": {
            "offset": 212837,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213900,
            "length": 13
          },
          "column_index": {
            "offset": 212865,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213913,
            "length": 13
          },
          "column_index": {
            "offset": 212901,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213926,
            "length": 13
          },
          "column_index": {
            "offset": 212929,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        }
      ],
      "row_count": 1024
    },
    {
      "columns": [
        {
          "path": "float_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213939,
            "length": 12
          },
          "column_index": {
            "offset": 212965,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "double_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213951,
            "length": 12
          },
          "column_index": {
            "offset": 212993,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "float_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213963,
            "length": 12
          },
          "column_index": {
            "offset": 213029,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213975,
            "length": 12
          },
          "column_index": {
            "offset": 213057,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213987,
            "length": 12
          },
          "column_index": {
            "offset": 213093,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 213999,
            "length": 12
          },
          "column_index": {
            "offset": 213121,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214011,
            "length": 13
          },
          "column_index": {
            "offset": 213157,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214024,
            "length": 13
          },
          "column_index": {
            "offset": 213185,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        }
      ],
      "row_count": 1024
    },
    {
      "columns": [
        {
          "path": "float_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214037,
            "length": 13
          },
          "column_index": {
            "offset": 213221,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "double_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214050,
            "length": 13
          },
          "column_index": {
            "offset": 213249,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "float_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214063,
            "length": 13
          },
          "column_index": {
            "offset": 213285,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214076,
            "length": 13
          },
          "column_index": {
            "offset": 213313,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214089,
            "length": 13
          },
          "column_index": {
            "offset": 213349,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214102,
            "length": 13
          },
          "column_index": {
            "offset": 213377,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214115,
            "length": 13
          },
          "column_index": {
            "offset": 213413,
            "length": 28
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214128,
            "length": 13
          },
          "column_index": {
            "offset": 213441,
            "length": 36
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        }
      ],
      "row_count": 808
    },
    {
      "columns": [
        {
          "path": "float_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214141,
            "length": 13
          },
          "column_index": {
            "offset": 213477,
            "length": 27
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "double_plain",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214154,
            "length": 13
          },
          "column_index": {
            "offset": 213504,
            "length": 35
          },
          "bloom_filter": null,
          "compression": "ZSTD",
          "encodings": [
            "PLAIN",
            "RLE"
          ]
        },
        {
          "path": "float_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214167,
            "length": 13
          },
          "column_index": {
            "offset": 213539,
            "length": 27
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_1024",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214180,
            "length": 13
          },
          "column_index": {
            "offset": 213566,
            "length": 35
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214193,
            "length": 13
          },
          "column_index": {
            "offset": 213601,
            "length": 27
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_4096",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214206,
            "length": 13
          },
          "column_index": {
            "offset": 213628,
            "length": 35
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "float_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214219,
            "length": 13
          },
          "column_index": {
            "offset": 213663,
            "length": 27
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        },
        {
          "path": "double_alp_32",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": false,
          "offset_index": {
            "offset": 214232,
            "length": 13
          },
          "column_index": {
            "offset": 213690,
            "length": 35
          },
          "bloom_filter": null,
          "compression": "UNCOMPRESSED",
          "encodings": [
            "RLE",
            "ALP"
          ]
        }
      ],
      "row_count": 32
    }
  ],
  "footer": {
    "metadata_size": 4747
  }
}

@alamb

alamb commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

I have verified that the Rust reader from @sdf-jkl in apache/arrow-rs#9372 can read this file correctly

@alamb
alamb marked this pull request as ready for review August 4, 2026 14:24
@alamb alamb changed the title Add ALP test data Add single ALP test file (211KB, covers corner cases) Aug 4, 2026
@vinooganesh

Copy link
Copy Markdown

I just ran this against the Java implementation from apache/parquet-java#3397. It reads the file fine, no changes needed on our end.

I decoded all six ALP columns and compared them against the matching PLAIN columns as raw bits (doubleToRawLongBits / floatToRawIntBits, so NaN and -0.0 compare properly). 53,952 values, everything matched. The special values at 1024-2047 all survive: both NaNs, +/-Inf, -0.0 at 2002, and the subnormal at 2003. The constant vector in row group 2 and the 8 nulls in the trailing partial vector are fine too.

It's worth calling out that the _32 columns caught something my own fixtures didn't. Java accepts log_vector_size 3-15, but everything I'd generated was 1024 or 4096, so vector size 32 had never actually been decoded from another writer's file until now. It works, but that path was untested.

There are a couple of things I noticed while poking at it, and all of them are fixable with a few extra columns rather than more data:

  • Every data page in the file is V1. Nothing exercises DataPageV2.
  • All the NaNs are canonical (0x7FF8000000000000 / 0x7FC00000), so nothing checks that a reader preserves NaN payloads.
  • The widest FOR bit width in the file is 18. @emkornfield asked us to cover values needing 63/64 bits after FOR on the Java PR, and that case isn't here.

I have fixtures for all three if they'd be helpful, but they'd be more columns on this file rather than separate ones. And I'm happy to contribute the Java-side conformance test to parquet-java once this lands on master.

@alamb

alamb commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

I have fixtures for all three if they'd be helpful, but they'd be more columns on this file rather than separate ones. And I'm happy to contribute the Java-side conformance test to parquet-java once this lands on master.

Thanks @vinooganesh

Every data page in the file is V1. Nothing exercises DataPageV2.

I think this is ok as I don't think the ALP encoding changes with the datapage format (and thus I don't think we need to cover that dimension in the ALP test cases)

All the NaNs are canonical (0x7FF8000000000000 / 0x7FC00000), so nothing checks that a reader preserves NaN payloads.

This is a good point -- I could update the file to also contain some non canonical Nans

The widest FOR bit width in the file is 18. @emkornfield asked us to cover values needing 63/64 bits after FOR on the Java PR, and that case isn't here.

This is a good point too -- I think I could update this file so that one of the groups has larger magnitude values that would use a large FOR.

I will wait for a few more days to see if there are other suggestions / comments about this PR, and then update the file per your suggestions

@alamb alamb changed the title Add single ALP test file (211KB, covers corner cases) Add single ALP test file (216KB, covers corner cases) Aug 7, 2026
@alamb

alamb commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

@vinooganesh I have updated the generator here to add some new Nan values and a small new row group that has a large FOR (18446744073709550676 for double)
apache/arrow@195ebd5

CurtHagenlocher added a commit to CurtHagenlocher/parquet-testing that referenced this pull request Aug 13, 2026
FSST (parquet-format#531) is in its final phases and needs working
implementations to validate that implementers read the wire format the same
way. These files are one implementation's output, offered for that purpose, in
the same spirit as the ALP files in apache#100 and apache#119.

Two files, identical data, differing only in compression: the symbol table page
carries its own is_compressed flag, so a reader that ignores it passes one file
and fails the other.

Six columns per row group, all holding the same values: a reference pair encoded
with DELTA_LENGTH_BYTE_ARRAY, an FSST pair (8-bit codes) and an FSST_16 pair
(16-bit codes). A decoder bit-compares the FSST columns against the reference
columns, so the files need no expected data alongside them.

Four row groups, each training its own symbol table: URLs, NULLs and empty
strings, escape-forcing values, and long values. Across the file that gives 104
data pages, 88 with DELTA_BINARY_PACKED offset arrays and 16 with PLAIN, and 110
escape sequences.

The README records the two places the file reflects a reading of the proposal
rather than its explicit text — the PageHeader field id for
symbol_table_page_header, and the FSST_16 escape framing — so that a diverging
implementation knows where to look. It also records the one gap: no FSST_16
column contains an escape, because the generating implementation's 16-bit tables
always hold all 256 single-byte symbols.
CurtHagenlocher added a commit to clast-project/engineered-wood that referenced this pull request Aug 13, 2026
#153)

* feat(testtool): generate FSST reference data for other implementations

Adds `create_fsst_test_file`, which writes one small Parquet file exercising
the FSST encoding's corner cases, modelled on the ALP file in
apache/parquet-testing#119: one file rather than a corpus, and every FSST
column duplicated by a DELTA_LENGTH_BYTE_ARRAY column holding the same values,
so a reader can bit-compare the two without any expected data alongside the
file.

The corpus is synthetic and seeded, so the file is reproducible and carries no
third-party licensing. Four row groups, each training its own symbol table:
URLs, nulls-and-empties, escape-forcing values, and long values.

The command reads the file back before reporting, because most of what makes it
useful is chosen rather than declared — the writer picks the offset encoding per
page, the trainer picks the symbol table, and §7.5 may decline FSST for a chunk
entirely. Two rounds of that feedback changed the corpus:

- The escape section produced no escapes at all. It put a rare code point in
  every value, so the trainer simply learned it as a symbol. Only every ninth
  value carries one now, and the file has 78 escape sequences.
- A binary chunk fell back under §7.5 and vanished from the summary, which read
  as a bug in the summary rather than the writer declining. Fallback is now
  reported explicitly.

Also shrank the file from 520 KB to 182 KB uncompressed (80 KB with --compression
zstd), since parquet-testing#119 rejected its predecessor for size.

Grants Core internals to the TestTool, which every other consuming project
already had, so the page walk can decompress a compressed values section.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(testtool): three defects in the FSST generator found by review

The binary escape corpus overwrote the last byte of its own prefix. The
overwrite index was hardcoded to 24 against a 25-byte prefix, so it ate the
trailing '-'. Deriving the offset from the prefix instead is not cosmetic: it
lifts the file from 110 escape sequences to 181, and the binary FSST column in
the escapes row group goes from 0 escapes to 2 — that column was not exercising
the escape path at all.

DescribeTable printed `headerSize + body.Length - headerSize`, which is just
body.Length, leaving headerSize computed and unused. The whole serialized body
is the right thing to report, so the number is unchanged; the dead term is gone
and the label is now explained.

--compression parsed the entire CompressionCodec enum, so LZO — which the writer
cannot emit — was accepted and failed later as a NotSupportedException from
inside the page writer. Validates against the codecs the writer supports and
fails with a usage error naming them.

Found by Copilot review on #153.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment thread data/README.md
| Rows | Contents | Rationale / coverage |
|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0–1023 | base | Happy path: full vector, small frame-of-reference bit width, no exceptions |
| 1024–2047 | base, plus: NaN at 1024, 1500 and 2047 (three distinct bit patterns, see below), +Inf at 2000, −Inf at 2001, −0.0 at 2002, subnormal (`5e-324` double / `1e-45` float) at 2003 | NaN / Inf and sign/precision edge values via the exception mechanism; exceptions at exact vector boundaries; NaN payload preservation; statistics conventions (NaN excluded from min/max, ±Inf included, `nan_count`, −0.0/+0.0 normalization) |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nan_count is unset in the file, Is that intended?

Comment thread data/README.md

To check conformance of an `ALP` decoder, read each `ALP`-encoded column and
compare the decoded values against the values from the corresponding
`PLAIN`-encoded column. The values should be match exactly (bitwise).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`PLAIN`-encoded column. The values should be match exactly (bitwise).
`PLAIN`-encoded column. The values should match exactly (bitwise).

Comment thread data/README.md
(`ALP = 10`).
It was created with the code in this [PR](https://github.com/apache/arrow/pull/49154).

All columns contain the same 9032 values. The same values appear all columns so

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
All columns contain the same 9032 values. The same values appear all columns so
All columns contain the same 9032 values. The same values appear in all columns so

Comment thread data/README.md
| 5120–6143 | all values full-mantissa random | All exceptions |
| 6144–7167 | base but with 4 decimal digits (e.g. `3.1416`) | Different exponent/factor than the other vectors |
| 7168–8191 | constant (all `7.77`) | `bit_width = 0` vectors |
| 8192–8999 | base, with nulls at every 100th row (8 nulls) | Partial vector + null handling |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Slightly different in the file I think, it's 8200, 8300, 8400, 8500, 8600, 8700, 8800, 8900 are null. So every row ordinal divisible by 100 is null.

Again just a nit in the wording, since I assumed 100th row meant 8292, 8392 ...

@RussellSpitzer RussellSpitzer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested against the C++ and Java impls as well, both worked fine. Left a few nits on wording but this looks good to me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add example ALP files

4 participants