Skip to content

Antalya 26.6: Support Nullable(Tuple) for Arrow, ArrowStream, ORC, legacy Parquet formats - #2128

Open
zvonand wants to merge 2 commits into
antalya-26.6from
feature/antalya-26.6/pr-1802
Open

Antalya 26.6: Support Nullable(Tuple) for Arrow, ArrowStream, ORC, legacy Parquet formats#2128
zvonand wants to merge 2 commits into
antalya-26.6from
feature/antalya-26.6/pr-1802

Conversation

@zvonand

@zvonand zvonand commented Jul 29, 2026

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Support Nullable(Tuple) for Arrow, ArrowStream, ORC, legacy Parquet formats (ClickHouse#101272 by @nihalzp) (#1802 by @zvonand).

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

Cherry-picked from #1802.

zvonand added 2 commits July 29, 2026 18:44
@zvonand
Cherry-pick of #1802 with unresolved conflict markers (resolution in …
66af140 
…next commit)

---
Original cherry-pick message follows:

Merge pull request #1802 from Altinity/feature/antalya-26.3/ClickHouse-ClickHouse-pr-101272

Antalya 26.3: Support Nullable(Tuple) for Arrow, ArrowStream, ORC, legacy Parquet formats
# Conflicts:
#	src/Processors/Formats/Impl/ArrowColumnToCHColumn.cpp
#	src/Processors/Formats/Impl/CHColumnToArrowColumn.cpp
#	src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp
#	tests/queries/0_stateless/04065_tuple_inside_nullable_parquet_roundtrip.reference
#	tests/queries/0_stateless/04065_tuple_inside_nullable_parquet_roundtrip.sql
@zvonand
Resolve conflicts in cherry-pick of #1802
82b83ec 
All conflicts resolved in favour of "ours" (antalya-26.6): the whole
content of PR #1802 (Nullable(Tuple) support for Arrow/ArrowStream/ORC/
legacy Parquet) is already present on antalya-26.6:

* Nested::unwrapNullableTuple (src/DataTypes/NestedUtils.{h,cpp}), the
  insertNullAsDefaultIfNeeded recursion and the ORC/CSV changes applied
  as no-ops (identical content already on the branch).
* ArrowColumnToCHColumn.cpp: readColumnWithNumericData already zeroes
  values at null positions, readNonNullableColumnFromArrowColumn already
  routes Nested list elements through Nested::unwrapNullableTuple and the
  arrow::Type::STRUCT exclusion is already removed. Kept the branch's
  extra ColumnArray last-offset validation; dropped the cherry-pick's
  stray copy of the null-zeroing loop that git mis-applied to
  readColumnWithDurationData (that hunk belongs to readColumnWithNumericData
  in the source PR and is already there).
* CHColumnToArrowColumn.cpp: the branch's fillArrowArray returns an
  arrow::Array and builds the struct via arrow::StructArray::Make with a
  null bitmap; it already implements the source PR's semantics (children
  get nullptr instead of the struct-level bytemap, struct-level nulls come
  from the bytemap). The cherry-pick's builder.Append()/AppendNull() form
  targets the old, pre-refactor signature.
* MsgPackRowInputFormat.cpp: the Nullable unwrapping in start_array is
  already present, only placed after the tuple-size check.
* 04065_tuple_inside_nullable_parquet_roundtrip.{sql,reference}: the
  branch version was deliberately rewritten for the V3-only Parquet
  reader (commit 0f9384b), the obsolete setting
  input_format_parquet_use_native_reader_v3 no longer exists on
  antalya-26.6, so the cherry-pick's Arrow-reader variants are dropped.
@zvonand zvonand added releasy Created/managed by RelEasy antalya-26.6 ai-resolved Port conflict auto-resolved by Claude labels Jul 29, 2026
@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown

Workflow [PR], commit [82b83ec]

@zvonand

zvonand commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@blau-ai

@blau-ai

blau-ai commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

CI triage for #2128

Verdict: 9 failing regression jobs — 0 PR-caused. All are pre-existing on the base branch or infrastructure/flake. There is nothing here that this PR broke. The bigger thing to flag is that the build + functional + integration suites were all skipped, so this PR's actual code was never compiled or tested by CI (details at the bottom).

Baseline used for comparison: latest antalya-26.6 MasterCI run 30468399078 (2026-07-29, immediately before this PR).

Per-failure

Check (aarch64 + release) Class Evidence
Iceberg (1) / (2) Pre-existing Iceberg (1)/(2) fail identically on the base run 30468399078.
Swarms Pre-existing + env Mass failure (1334/1520 scenarios) — that pattern is environmental, not a targeted regression. Also fails on base.
S3Export (partition) Pre-existing S3Export (partition) fails on the base run too.
S3Export (part) Infra / timeout Ran ~3h30m then cancelled (job timeout). Also cancelled on base — not PR-specific.
ParquetS3 (aws_s3), aarch64 Infra (dead runner) Only new-vs-base failure, but the check annotation says: "The self-hosted runner lost communication with the server…" — not a Parquet assertion.

Why the ParquetS3 one is not the PR (even though the PR touches Parquet)

This is the only job that was green on base and red here, so it deserved a close look — but:

  • It failed with a runner-lost-communication annotation, not a test assertion.
  • ParquetS3 (aws_s3) on release passed, and ParquetS3 (minio) passed on both arches. A real regression in the Parquet reader/writer code would show up on minio and on release too, not on a single aarch64/aws_s3 runner. The aws_s3 variant hits real AWS S3 and is a routine flake source.

Fix: none needed — just re-run ParquetS3 (aws_s3) / aarch64. The rest are pre-existing base-branch breakage that this PR is not responsible for (worth a separate infra/flaky-suite ticket if not already tracked).

⚠️ The more important issue: this PR wasn't actually built or functionally tested

On both CI runs for this commit — 30475132112 and the companion 30475129123 — every Build, Fast test, Stateless tests, and Integration tests job is skipped (140+ skipped jobs; only regression suites + bookkeeping ran). Consequences:

  • The C++ changes in ArrowColumnToCHColumn.cpp, CHColumnToArrowColumn.cpp, MsgPackRowInputFormat.cpp were not compiled by CI.
  • The new stateless test 04065_tuple_inside_nullable_parquet_roundtrip (the one that directly exercises the Nullable(Tuple) round-trip this PR adds) was never executed.
  • The failing regression suites (Iceberg/Swarms/S3Export) don't meaningfully cover the Nullable(Tuple) Arrow/ORC/Parquet path, so a green-except-these result would still tell you nothing about whether the feature works.

Recommended next step: trigger a full CI run (builds + Fast test + Stateless + Integration) so the changes are compiled and 04065_tuple_inside_nullable_parquet_roundtrip actually runs. That — not the regression suites above — is what will validate this PR. The build appears skipped likely because the workflow treated this as a "Community PR"; a maintainer re-run/label should enable the full pipeline.

Note: I can read CI status and logs but cannot build or run ClickHouse in this container, and the testflows report HTML on S3 isn't fetchable from here — classifications above are based on GitHub check annotations, job conclusions, and the base-branch baseline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ai-resolved Port conflict auto-resolved by Claude antalya-26.6 releasy Created/managed by RelEasy

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@zvonand @blau-ai