[common][spark] Fix ClassCastException when reading a VECTOR column as ARRAY#8830
Merged
JingsongLi merged 3 commits intoJul 23, 2026
Merged
Conversation
A VECTOR column is exposed as ARRAY at the engine boundary (e.g. Flink maps VECTOR to ARRAY), so it is read through VectorizedColumnBatch.getArray. But the columnar read produces a VecColumnVector (e.g. CastedVectorColumnVector), and getArray blindly cast it to ArrayColumnVector, throwing ClassCastException. Since a vector is an array (InternalVector extends InternalArray), read a VecColumnVector column via getVector, mirroring what ColumnarArray and RowToColumnConverter already do for the ARRAY-vs-VECTOR split.
For ARRAY<VECTOR<..>> / MAP<.., VECTOR<..>>, the element column is a VecColumnVector read through ColumnarArray.getArray, which had the same blind cast to ArrayColumnVector and threw ClassCastException. Apply the same branch, and add a nested regression test.
For ARRAY<VECTOR<..>> / MAP<.., VECTOR<..>>, a Spark array element's type is a VectorType (VECTOR is exposed as a Spark array), but SparkArrayData.getArray cast the element type to ArrayType unconditionally, throwing ClassCastException. Branch on VectorType vs ArrayType, mirroring AbstractSparkInternalRow.getArray.
XiaoHongbo-Hope
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Reading a
VECTORcolumn through Flink throws:Flink has no vector type, so
DataTypeToLogicalTypemaps a PaimonVectorTypeto a FlinkArrayType, and Flink's generated code reads the column viagetArray. However the columnar read produces aVecColumnVector(e.g.CastedVectorColumnVector/ReadableVectorColumnVector), andVectorizedColumnBatch.getArrayunconditionally cast it toArrayColumnVector, causing theClassCastException.(Spark is not affected: it keeps the Paimon
VectorTypeand reads a VECTOR column viagetVector, seeAbstractSparkInternalRow. Only the Flink path, which flattens VECTOR to ARRAY and callsgetArray, hits this.)Since a vector is an array (
InternalVector extends InternalArray), read aVecColumnVectorcolumn throughgetVector. This mirrors the ARRAY-vs-VECTOR branching thatColumnarArrayandRowToColumnConverteralready do, and makesgetArrayrobust for any caller that reads a VECTOR column as an array.Tests
ColumnarRowWithVectorTest#testVectorReadAsArray— reads a VECTOR column viagetArray; fails withClassCastExceptionbefore the fix, passes after.