Add single ALP test file (216KB, covers corner cases) - #119
Conversation
| The same values appear in `PLAIN`-encoded reference columns so decoders can | ||
| bit-compare the `ALP` columns against known results: | ||
|
|
||
| | Column | Encoding | Rationale / coverage | |
There was a problem hiding this comment.
I tried to explain both what is in the file, as well as to explain the rationale for including each piece
There was a problem hiding this comment.
You can verify this a bit with standard tools. For example, using the tools from the Rust implementation in apache/arrow-rs#9372
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ ~/Software/arrow-rs/target/release/parquet-schema data/alp_extended.zstd.parquet
Metadata for file: data/alp_extended.zstd.parquet
version: 2
num of rows: 9000
created by: parquet-cpp-arrow version 23.0.0-SNAPSHOT
message schema {
OPTIONAL FLOAT float_plain;
OPTIONAL DOUBLE double_plain;
OPTIONAL FLOAT float_alp_1024;
OPTIONAL DOUBLE double_alp_1024;
OPTIONAL FLOAT float_alp_4096;
OPTIONAL DOUBLE double_alp_4096;
OPTIONAL FLOAT float_alp_32;
OPTIONAL DOUBLE double_alp_32;
}There was a problem hiding this comment.
The layout:
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ ~/Software/arrow-rs/target/release/parquet-layout data/alp_extended.zstd.parquetDetails
{
"row_groups": [
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211391,
"length": 12
},
"column_index": {
"offset": 210367,
"length": 28
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211403,
"length": 14
},
"column_index": {
"offset": 210395,
"length": 36
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211417,
"length": 14
},
"column_index": {
"offset": 210431,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211431,
"length": 14
},
"column_index": {
"offset": 210459,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211445,
"length": 14
},
"column_index": {
"offset": 210495,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211459,
"length": 14
},
"column_index": {
"offset": 210523,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211473,
"length": 14
},
"column_index": {
"offset": 210559,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211487,
"length": 14
},
"column_index": {
"offset": 210587,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 6144
},
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211501,
"length": 13
},
"column_index": {
"offset": 210623,
"length": 28
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211514,
"length": 13
},
"column_index": {
"offset": 210651,
"length": 36
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211527,
"length": 13
},
"column_index": {
"offset": 210687,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211540,
"length": 13
},
"column_index": {
"offset": 210715,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211553,
"length": 13
},
"column_index": {
"offset": 210751,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211566,
"length": 13
},
"column_index": {
"offset": 210779,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211579,
"length": 13
},
"column_index": {
"offset": 210815,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211592,
"length": 13
},
"column_index": {
"offset": 210843,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 1024
},
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211605,
"length": 12
},
"column_index": {
"offset": 210879,
"length": 28
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211617,
"length": 12
},
"column_index": {
"offset": 210907,
"length": 36
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211629,
"length": 12
},
"column_index": {
"offset": 210943,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211641,
"length": 12
},
"column_index": {
"offset": 210971,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211653,
"length": 12
},
"column_index": {
"offset": 211007,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211665,
"length": 12
},
"column_index": {
"offset": 211035,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211677,
"length": 13
},
"column_index": {
"offset": 211071,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211690,
"length": 13
},
"column_index": {
"offset": 211099,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 1024
},
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211703,
"length": 13
},
"column_index": {
"offset": 211135,
"length": 28
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211716,
"length": 13
},
"column_index": {
"offset": 211163,
"length": 36
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211729,
"length": 13
},
"column_index": {
"offset": 211199,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211742,
"length": 13
},
"column_index": {
"offset": 211227,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211755,
"length": 13
},
"column_index": {
"offset": 211263,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211768,
"length": 13
},
"column_index": {
"offset": 211291,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211781,
"length": 13
},
"column_index": {
"offset": 211327,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211794,
"length": 13
},
"column_index": {
"offset": 211355,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 808
}
],
"footer": {
"metadata_size": 3866
}
}There was a problem hiding this comment.
Updated layout
Details
{
"row_groups": [
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213725,
"length": 12
},
"column_index": {
"offset": 212453,
"length": 28
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213737,
"length": 14
},
"column_index": {
"offset": 212481,
"length": 36
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213751,
"length": 14
},
"column_index": {
"offset": 212517,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213765,
"length": 14
},
"column_index": {
"offset": 212545,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213779,
"length": 14
},
"column_index": {
"offset": 212581,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213793,
"length": 14
},
"column_index": {
"offset": 212609,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213807,
"length": 14
},
"column_index": {
"offset": 212645,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213821,
"length": 14
},
"column_index": {
"offset": 212673,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 6144
},
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213835,
"length": 13
},
"column_index": {
"offset": 212709,
"length": 28
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213848,
"length": 13
},
"column_index": {
"offset": 212737,
"length": 36
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213861,
"length": 13
},
"column_index": {
"offset": 212773,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213874,
"length": 13
},
"column_index": {
"offset": 212801,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213887,
"length": 13
},
"column_index": {
"offset": 212837,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213900,
"length": 13
},
"column_index": {
"offset": 212865,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213913,
"length": 13
},
"column_index": {
"offset": 212901,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213926,
"length": 13
},
"column_index": {
"offset": 212929,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 1024
},
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213939,
"length": 12
},
"column_index": {
"offset": 212965,
"length": 28
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213951,
"length": 12
},
"column_index": {
"offset": 212993,
"length": 36
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213963,
"length": 12
},
"column_index": {
"offset": 213029,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213975,
"length": 12
},
"column_index": {
"offset": 213057,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213987,
"length": 12
},
"column_index": {
"offset": 213093,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 213999,
"length": 12
},
"column_index": {
"offset": 213121,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214011,
"length": 13
},
"column_index": {
"offset": 213157,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214024,
"length": 13
},
"column_index": {
"offset": 213185,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 1024
},
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214037,
"length": 13
},
"column_index": {
"offset": 213221,
"length": 28
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214050,
"length": 13
},
"column_index": {
"offset": 213249,
"length": 36
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214063,
"length": 13
},
"column_index": {
"offset": 213285,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214076,
"length": 13
},
"column_index": {
"offset": 213313,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214089,
"length": 13
},
"column_index": {
"offset": 213349,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214102,
"length": 13
},
"column_index": {
"offset": 213377,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214115,
"length": 13
},
"column_index": {
"offset": 213413,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214128,
"length": 13
},
"column_index": {
"offset": 213441,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 808
},
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214141,
"length": 13
},
"column_index": {
"offset": 213477,
"length": 27
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214154,
"length": 13
},
"column_index": {
"offset": 213504,
"length": 35
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214167,
"length": 13
},
"column_index": {
"offset": 213539,
"length": 27
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214180,
"length": 13
},
"column_index": {
"offset": 213566,
"length": 35
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214193,
"length": 13
},
"column_index": {
"offset": 213601,
"length": 27
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214206,
"length": 13
},
"column_index": {
"offset": 213628,
"length": 35
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214219,
"length": 13
},
"column_index": {
"offset": 213663,
"length": 27
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 214232,
"length": 13
},
"column_index": {
"offset": 213690,
"length": 35
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 32
}
],
"footer": {
"metadata_size": 4747
}
}|
I have verified that the Rust reader from @sdf-jkl in apache/arrow-rs#9372 can read this file correctly |
|
I just ran this against the Java implementation from apache/parquet-java#3397. It reads the file fine, no changes needed on our end. I decoded all six ALP columns and compared them against the matching PLAIN columns as raw bits ( It's worth calling out that the There are a couple of things I noticed while poking at it, and all of them are fixable with a few extra columns rather than more data:
I have fixtures for all three if they'd be helpful, but they'd be more columns on this file rather than separate ones. And I'm happy to contribute the Java-side conformance test to parquet-java once this lands on master. |
Thanks @vinooganesh
I think this is ok as I don't think the ALP encoding changes with the datapage format (and thus I don't think we need to cover that dimension in the ALP test cases)
This is a good point -- I could update the file to also contain some non canonical Nans
This is a good point too -- I think I could update this file so that one of the groups has larger magnitude values that would use a large FOR. I will wait for a few more days to see if there are other suggestions / comments about this PR, and then update the file per your suggestions |
|
@vinooganesh I have updated the generator here to add some new Nan values and a small new row group that has a large FOR ( |
FSST (parquet-format#531) is in its final phases and needs working implementations to validate that implementers read the wire format the same way. These files are one implementation's output, offered for that purpose, in the same spirit as the ALP files in apache#100 and apache#119. Two files, identical data, differing only in compression: the symbol table page carries its own is_compressed flag, so a reader that ignores it passes one file and fails the other. Six columns per row group, all holding the same values: a reference pair encoded with DELTA_LENGTH_BYTE_ARRAY, an FSST pair (8-bit codes) and an FSST_16 pair (16-bit codes). A decoder bit-compares the FSST columns against the reference columns, so the files need no expected data alongside them. Four row groups, each training its own symbol table: URLs, NULLs and empty strings, escape-forcing values, and long values. Across the file that gives 104 data pages, 88 with DELTA_BINARY_PACKED offset arrays and 16 with PLAIN, and 110 escape sequences. The README records the two places the file reflects a reading of the proposal rather than its explicit text — the PageHeader field id for symbol_table_page_header, and the FSST_16 escape framing — so that a diverging implementation knows where to look. It also records the one gap: no FSST_16 column contains an escape, because the generating implementation's 16-bit tables always hold all 256 single-byte symbols.
#153) * feat(testtool): generate FSST reference data for other implementations Adds `create_fsst_test_file`, which writes one small Parquet file exercising the FSST encoding's corner cases, modelled on the ALP file in apache/parquet-testing#119: one file rather than a corpus, and every FSST column duplicated by a DELTA_LENGTH_BYTE_ARRAY column holding the same values, so a reader can bit-compare the two without any expected data alongside the file. The corpus is synthetic and seeded, so the file is reproducible and carries no third-party licensing. Four row groups, each training its own symbol table: URLs, nulls-and-empties, escape-forcing values, and long values. The command reads the file back before reporting, because most of what makes it useful is chosen rather than declared — the writer picks the offset encoding per page, the trainer picks the symbol table, and §7.5 may decline FSST for a chunk entirely. Two rounds of that feedback changed the corpus: - The escape section produced no escapes at all. It put a rare code point in every value, so the trainer simply learned it as a symbol. Only every ninth value carries one now, and the file has 78 escape sequences. - A binary chunk fell back under §7.5 and vanished from the summary, which read as a bug in the summary rather than the writer declining. Fallback is now reported explicitly. Also shrank the file from 520 KB to 182 KB uncompressed (80 KB with --compression zstd), since parquet-testing#119 rejected its predecessor for size. Grants Core internals to the TestTool, which every other consuming project already had, so the page walk can decompress a compressed values section. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(testtool): three defects in the FSST generator found by review The binary escape corpus overwrote the last byte of its own prefix. The overwrite index was hardcoded to 24 against a 25-byte prefix, so it ate the trailing '-'. Deriving the offset from the prefix instead is not cosmetic: it lifts the file from 110 escape sequences to 181, and the binary FSST column in the escapes row group goes from 0 escapes to 2 — that column was not exercising the escape path at all. DescribeTable printed `headerSize + body.Length - headerSize`, which is just body.Length, leaving headerSize computed and unused. The whole serialized body is the right thing to report, so the number is unchanged; the dead term is gone and the label is now explained. --compression parsed the entire CompressionCodec enum, so LZO — which the writer cannot emit — was accepted and failed later as a NotSupportedException from inside the page writer. Validates against the codecs the writer supports and fails with a usage error naming them. Found by Copilot review on #153. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
| | Rows | Contents | Rationale / coverage | | ||
| |-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | 0–1023 | base | Happy path: full vector, small frame-of-reference bit width, no exceptions | | ||
| | 1024–2047 | base, plus: NaN at 1024, 1500 and 2047 (three distinct bit patterns, see below), +Inf at 2000, −Inf at 2001, −0.0 at 2002, subnormal (`5e-324` double / `1e-45` float) at 2003 | NaN / Inf and sign/precision edge values via the exception mechanism; exceptions at exact vector boundaries; NaN payload preservation; statistics conventions (NaN excluded from min/max, ±Inf included, `nan_count`, −0.0/+0.0 normalization) | |
There was a problem hiding this comment.
nan_count is unset in the file, Is that intended?
|
|
||
| To check conformance of an `ALP` decoder, read each `ALP`-encoded column and | ||
| compare the decoded values against the values from the corresponding | ||
| `PLAIN`-encoded column. The values should be match exactly (bitwise). |
There was a problem hiding this comment.
| `PLAIN`-encoded column. The values should be match exactly (bitwise). | |
| `PLAIN`-encoded column. The values should match exactly (bitwise). |
| (`ALP = 10`). | ||
| It was created with the code in this [PR](https://github.com/apache/arrow/pull/49154). | ||
|
|
||
| All columns contain the same 9032 values. The same values appear all columns so |
There was a problem hiding this comment.
| All columns contain the same 9032 values. The same values appear all columns so | |
| All columns contain the same 9032 values. The same values appear in all columns so |
| | 5120–6143 | all values full-mantissa random | All exceptions | | ||
| | 6144–7167 | base but with 4 decimal digits (e.g. `3.1416`) | Different exponent/factor than the other vectors | | ||
| | 7168–8191 | constant (all `7.77`) | `bit_width = 0` vectors | | ||
| | 8192–8999 | base, with nulls at every 100th row (8 nulls) | Partial vector + null handling | |
There was a problem hiding this comment.
Slightly different in the file I think, it's 8200, 8300, 8400, 8500, 8600, 8700, 8800, 8900 are null. So every row ordinal divisible by 100 is null.
Again just a nit in the wording, since I assumed 100th row meant 8292, 8392 ...
RussellSpitzer
left a comment
There was a problem hiding this comment.
Tested against the C++ and Java impls as well, both worked fine. Left a few nits on wording but this looks good to me
(Draft until I validate this example some more)
Rationale
#100 from @prtkgaur and @vinooganesh at prtkgaur#1 have coverage but are also 4x larger than all the existing data files
Thus I want to create an example file that can be used by readers but is not large and still covers all the corner cases
Changes
full file layout details