Skip to content

Compute paired SciPy distances row-wise - #136

Draft
afermg wants to merge 1 commit into
mainfrom
perf/rowwise-pair-metrics
Draft

Compute paired SciPy distances row-wise#136
afermg wants to merge 1 commit into
mainfrom
perf/rowwise-pair-metrics

Conversation

@afermg

@afermg afermg commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

Replace the full B × B scipy.spatial.distance.cdist allocation used only to read its diagonal with direct row-wise kernels for:

  • Hamming
  • Jaccard
  • squared Euclidean
  • Euclidean
  • city-block / Manhattan

Unsupported metrics and callables retain the SciPy fallback. Public batched similarity results remain float32; the private helper retains SciPy's float64 boundary.

Benchmark

Synthetic paired rows, 1,000 × 196, two repeats, median wall time:

Metric origin/main This PR Speedup
Hamming 3.653 s 0.000198 s 18,430×
Jaccard 6.156 s 0.000538 s 11,440×
squared Euclidean 2.432 s 0.000849 s 2,865×
city-block 5.310 s 0.000884 s 6,004×

Hamming and Jaccard outputs were byte-identical. Squared-Euclidean and city-block results matched within floating-point tolerance. At 4,000 × 196, Hamming peak RSS dropped from 235.5 MB to 102.2 MB while producing an identical digest.

The host was heavily contended, so absolute timings are noisy; the removal of quadratic allocation/work is deterministic.

Validation

  • 47 focused tests passed under SciPy 1.16.3 and cached SciPy 1.13.1
  • signed/unsigned, boolean, float, nonfinite, empty-shape, alias, callable, and public-dispatch coverage
  • Ruff and git diff --check passed
  • independent review: OK with non-blocking notes

@afermg
afermg marked this pull request as draft September 2, 2026 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant