Skip to content

[SM90] Add optimized SM90 support. - #14

Open
yuyu5333 wants to merge 1 commit into
deepseek-ai:mainfrom
yuyu5333:optimize/deepselect-tuning
Open

yuyu5333 wants to merge 1 commit into
deepseek-ai:mainfrom
yuyu5333:optimize/deepselect-tuning

Conversation

@yuyu5333

Copy link
Copy Markdown

Summary

This PR adds an optimized SM90 build and runtime path for Hopper GPUs such as H20 and H100, while preserving the existing SM100/SM103 defaults.

SM90 uses an 8-CTA cluster for small-batch, long-sequence BF16 Top-K workloads. The cluster path is enabled from a vocabulary size of 128K, based on measurements on an 8×H20 machine. SM100 and SM103 continue to use the existing 16-CTA cluster and 512K threshold.

Changes

  • Add DEEP_SELECT_CUDA_ARCHS to select 90a, 100a, and/or 103a at build time.
  • Keep the default build targets unchanged: sm_100a and sm_103a.
  • Add eight SM90 cluster instantiations covering:
    • int32 and int64 output indices
    • sorted_index enabled and disabled
    • return_value enabled and disabled
  • Dispatch SM90 BF16 workloads to an 8-CTA cluster when:
    • batch_size <= 6
    • vocab_size >= 128K
    • topk <= 1024
  • Preserve the existing 16-CTA cluster and 512K threshold for SM100/SM103.
  • Fall back to nvidia-smi -L for CUDA platform detection when lspci is unavailable in a container.
  • Avoid the std::format dependency in the C++ binding so the extension builds with older host standard libraries commonly found in CUDA containers.
  • Document the SM90 build command:
DEEP_SELECT_CUDA_ARCHS=90a pip install -v .

Performance

Measured on NVIDIA H20 with BF16 input, batch_size 1 and 6, and topk 512 and 1024. Each result is the median of 100 cold-cache runs.

The improvement column compares the new 8-CTA cluster with the existing single-CTA path compiled for SM90.

Vocabulary size Improvement over single CTA Speedup over torch.topk
128K 25.2%–35.0% 3.11×–3.83×
256K 36.5%–44.6% 2.79×–3.57×
512K 52.4%–56.1% 2.66×–3.75×
1M 61.5%–62.3% 2.12×–4.11×

Short-sequence and large-batch workloads continue to use the existing normal kernel. Their measured speedup over torch.topk remains 4.88×–6.27×.

Validation

  • Clean SM90 build with CUDA 13.0 and PyTorch 2.11.
  • cuobjdump register-spill check passed with no spills.
  • 20/20 targeted correctness cases passed.
  • Coverage included:
    • normal and 8-CTA cluster paths
    • BF16 and FP32 inputs
    • int32 and int64 indices
    • sorted_index
    • optional value output
    • variable row lengths through end
    • per-row index offsets
  • The final performance matrix passed Top-K value-set validation against torch.topk for every case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant