Skip to content

Reduce redundant Top-K host setup - #8

Open
morluto wants to merge 1 commit into
deepseek-ai:mainfrom
morluto:perf/host-launch-setup
Open

Reduce redundant Top-K host setup#8
morluto wants to merge 1 commit into
deepseek-ai:mainfrom
morluto:perf/host-launch-setup

Conversation

@morluto

@morluto morluto commented Sep 10, 2026

Copy link
Copy Markdown

Fixes #7.

Reduce repeated CPU setup before Top-K launches without changing device-side computation or caching tensor descriptors.

Cache the tensor-map encoder pointer using thread-safe local-static initialization, retrying lookup if initialization throws. Add a fixed-rank metadata path while preserving the vector helper interface. Route Top-K's rank-three metadata through stack arrays, and remove duplicate attribute setters from the three launchers; the common launcher still sets and checks the attributes for each invocation.

The host regression harness uses actual CUDA types and product headers with a recording mock resolver/encoder. It checks concurrent cold initialization, sharing across two translation units, lookup failure retries, encoder failure recovery, fresh per-call arguments, and vector/array equivalence for ranks 1–5. The same test against baseline headers fails the intended single-lookup assertion.

With 200,000 calls per round, the baseline metadata path makes 1,600,000 allocations and 200,000 lookups; the candidate makes zero of either after warmup. Across 14 rounds per variant in baseline/candidate/candidate/baseline order, median instrumented host preparation time was 457 ns versus 114 ns on an Intel i7-10875H using NVCC 13.3, -O2, and libstdc++. Allocation accounting and a mock encoder are included in these timings: they are not driver-encoding or end-to-end Top-K results. Reproduction commands are in tests/host/README.md.

Validation also includes SM100a compilation of one specialization from each of the BF16, FP32, and BF16-cluster kernel families, plus whitespace checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Top-K repeats driver lookup, metadata allocations, and launch attributes

1 participant