Skip to content

Optimize multilabel pair matching - #137

Draft
afermg wants to merge 2 commits into
mainfrom
perf/multilabel-matching
Draft

Optimize multilabel pair matching#137
afermg wants to merge 2 commits into
mainfrom
perf/multilabel-matching

Conversation

@afermg

@afermg afermg commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

Optimize find_pairs_multilabel by:

  • exploding labels into an inverted relation and equijoining positive pair-label rows
  • avoiding the previous all-row list_intersect cross product for positive matching
  • projecting only required columns in the negative disjointness query
  • preserving pandas and DuckDB relation inputs
  • quoting/escaping SQL identifiers
  • returning deterministic grouped positive associations

Negative disjointness still requires quadratic candidate comparisons; this PR reduces its projection/materialization cost rather than changing that algorithm.

Benchmark

Real JUMP-lite group_low: 1,342 profiles, 709 labels, 247,644 positive pair-label rows, and 834,814 negative pairs. Two-run medians:

Query origin/main This PR Speedup
Positive 16.072 s 0.193 s 83.1×
Negative 21.627 s 6.802 s 3.18×
Combined 37.699 s 6.996 s 5.39×

Peak process RSS dropped from 5.15 GB to 158 MB. Output cardinalities, label keys, counts, and pair associations matched; this PR deliberately makes pair ordering deterministic, so raw pair-array byte digests differ from the prior query-plan-dependent order.

Absolute times were noisy on a heavily contended host, but the SQL-plan and memory improvements were repeatable.

Validation

  • 20 focused multilabel tests passed
  • coverage includes pandas/DuckDB inputs, sameby/diffby combinations, repeated/null/empty labels, reserved/spaced/quoted identifiers, ordering, dtypes, and empty outputs
  • Ruff and git diff --check passed
  • independent follow-up review: OK

@afermg
afermg marked this pull request as draft September 2, 2026 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant