Reduce redundant Top-K host setup - #8
Open
morluto wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7.
Reduce repeated CPU setup before Top-K launches without changing device-side computation or caching tensor descriptors.
Cache the tensor-map encoder pointer using thread-safe local-static initialization, retrying lookup if initialization throws. Add a fixed-rank metadata path while preserving the vector helper interface. Route Top-K's rank-three metadata through stack arrays, and remove duplicate attribute setters from the three launchers; the common launcher still sets and checks the attributes for each invocation.
The host regression harness uses actual CUDA types and product headers with a recording mock resolver/encoder. It checks concurrent cold initialization, sharing across two translation units, lookup failure retries, encoder failure recovery, fresh per-call arguments, and vector/array equivalence for ranks 1–5. The same test against baseline headers fails the intended single-lookup assertion.
With 200,000 calls per round, the baseline metadata path makes 1,600,000 allocations and 200,000 lookups; the candidate makes zero of either after warmup. Across 14 rounds per variant in baseline/candidate/candidate/baseline order, median instrumented host preparation time was 457 ns versus 114 ns on an Intel i7-10875H using NVCC 13.3,
-O2, and libstdc++. Allocation accounting and a mock encoder are included in these timings: they are not driver-encoding or end-to-end Top-K results. Reproduction commands are intests/host/README.md.Validation also includes SM100a compilation of one specialization from each of the BF16, FP32, and BF16-cluster kernel families, plus whitespace checks.