Skip to content

[Performance] Optimize DataSetWithoutTimeGenerator merge hot path - #912

Closed
Caideyipi wants to merge 2 commits into
apache:developfrom
Caideyipi:optimize/data-set-without-time-generator-scan
Closed

[Performance] Optimize DataSetWithoutTimeGenerator merge hot path#912
Caideyipi wants to merge 2 commits into
apache:developfrom
Caideyipi:optimize/data-set-without-time-generator-scan

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

What changed:

DataSetWithoutTimeGenerator already scans every series for each emitted RowRecord. This change computes the next minimum timestamp during that required scan, replacing the additional boxed List/Boolean state, LongHeapPriorityQueue, and LongOpenHashSet maintenance.

The merge behavior is covered across sparse timestamps, batch boundaries, empty readers, duplicate timestamps, and Long.MIN_VALUE/Long.MAX_VALUE.

Validation:

  • mvnw.cmd test -P with-java -pl java/tsfile -Dtest=DataSetWithoutTimeGeneratorTest
  • 3 tests passed; Checkstyle and Spotless passed.
  • The SCADA benchmark identified DataSetWithoutTimeGenerator.timeHeapPut as 181 JFR samples spanning 6.533 seconds.

Manual performance benchmark (192.168.130.18, Linux, JDK 17.0.9; median ns/row; speedup = heap/set baseline / optimized scan):

Series Workload Optimized Heap/set baseline Speedup
64 aligned, 10,000 rows 1,155.8 1,078.7 0.93x
64 staggered, 32,000 rows 793.1 535.1 0.67x
128 aligned, 10,000 rows 2,396.0 2,457.7 1.03x
128 staggered, 64,000 rows 1,638.8 1,023.0 0.62x
256 aligned, 10,000 rows 4,835.3 5,207.8 1.08x
256 staggered, 128,000 rows 2,725.3 2,172.8 0.80x
64 aligned, 100,000 rows 1,317.1 1,105.8 0.84x
64 staggered, 320,000 rows 780.5 737.0 0.94x
128 aligned, 100,000 rows 2,818.2 2,356.3 0.84x
128 staggered, 640,000 rows 1,262.9 1,034.9 0.82x

All benchmark runs produced identical row counts and checksums for the optimized and baseline implementations. The measurements do not show a stable end-to-end performance improvement; the staggered workload is repeatedly slower. The benchmark is retained as a manually enabled diagnostic test (-Dtsfile.runPerformanceTests=true) and does not make machine-dependent timing assertions.

The branch is based on the latest upstream/develop and has only been pushed to the contributor fork.

@Caideyipi Caideyipi closed this Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant