ModelOpt 0.45.0 Release

New Features

Quantization

• Add NVFP4 W4A16 weight-only quantization (w4a16_nvfp4): FP4 weights with group_size=16, BF16 activations, no calibration forward pass required. Use mtq.W4A16_NVFP4_CFG or --qformat w4a16_nvfp4 in hf_ptq.py. vLLM deployment support is in progress.
• Add --cast_mxfp4_to_nvfp4 flag to examples/llm_ptq/hf_ptq.py for closed-form, bit-exact MXFP4 → NVFP4 weight conversion. Supports the GPT-OSS family (openai/gpt-oss-20b, openai/gpt-oss-120b). See examples/llm_ptq/README.md for usage.
• Add --cast_mxfp4_to_nvfp4 flag to examples/deepseek/deepseek_v4/quantize_to_nvfp4.py for closed-form, bit-exact MXFP4 → NVFP4 conversion of DeepSeek V4 routed-expert weights (mirrors the GPT-OSS cast; w1/w3 share one per-tensor scale_2 for the fused GEMM1). Activation input_scale still comes from --amax_path calibration.
• DeepSeek PTQ (examples/deepseek/ptq.py) now defaults to native top-k calibration with post-hoc per-layer peer-max sync of expert input_quantizer.amax; the all-experts path is preserved behind --calib_all_experts.
• Add active-MoE cost accounting for mtq.auto_quantize effective-bits search. Set constraints={"effective_bits": ..., "cost_model": "active_moe", "cost": {"active_moe_expert_ratio": ...}} to weight routed MoE expert costs by active experts per token while keeping shared experts fully counted. The hf_ptq.py AutoQuant path exposes this via --auto_quantize_cost_model active_moe and --auto_quantize_active_moe_expert_ratio.
• Add quantized nn.Embedding support. nn.Embedding is now registered in QuantModuleRegistry and exposes weight_quantizer (embedding table), output_quantizer (lookup activations), and a permanently disabled input_quantizer placeholder — embedding inputs are integer indices and cannot be fake-quantized, so direct enable*() calls raise. export_hf_checkpoint packs quantized embedding weights alongside Linear layers. Embedding quantizers are opt-in (parent_class: nn.Embedding disabled by default).
• Add composable $import system for recipe YAML configs, enabling reusable config snippets referenced via {$import: name} markers. All built-in PTQ recipes converted to use imports with shared snippets under modelopt_recipes/configs/ (numeric formats, quant_cfg building blocks, presets). See composable-imports docs.
• The PTQ example scripts examples/llm_ptq/hf_ptq.py, examples/llm_ptq/multinode_ptq.py and examples/megatron_bridge/quantize.py now derive their --qformat / --kv_cache_qformat (--quant_cfg / --kv_cache_quant for Megatron-Bridge) CLI vocabularies by discovering the YAML presets under modelopt_recipes/configs/ptq/presets/{model,kv}/ rather than carrying hardcoded QUANT_CFG_CHOICES / KV_QUANT_CFG_CHOICES tables. The discovery helper, alias table and ready-built QUANT_CFG_CHOICES / KV_QUANT_CFG_CHOICES mappings now live in modelopt.recipe.presets and are shared by all three scripts. Presets are loaded eagerly into a plain dict at import. Adding a new preset YAML makes it available on the CLI of all three with no script change — note this means each script now accepts every preset under those directories, not just a previously curated subset. All previously-supported short names (int8_sq, nvfp4_awq, fp8_pb_wo, nvfp4_mse, w4a8_awq, nvfp4_local_hessian, fp8_pc_pt, int8_wo) keep working via a small deprecation alias table; new formats should be exposed as preset YAMLs (or, longer term, as full --recipe recipes).
• Add configs/ptq/presets/kv/fp8_cast.yaml and configs/ptq/presets/kv/nvfp4_cast.yaml, promoting fp8_cast / nvfp4_cast to first-class KV presets composed from the existing kv_fp8_cast / kv_nvfp4_cast unit fragments. The previous runtime use_constant_amax post-edit in hf_ptq.py is removed; use_constant_amax: true now lives in the YAML and is therefore authoritative. Custom (out-of-tree) recipes that target a cast KV format must set use_constant_amax: true themselves on the [kv]_bmm_quantizer config — in-tree recipes already do via the kv_*_cast units.
• Add FP8 KV-cache cast variants for the partial-NVFP4 and weight-only general PTQ recipes: general/ptq/nvfp4_mlp_only-kv_fp8_cast, general/ptq/nvfp4_experts_only-kv_fp8_cast, general/ptq/nvfp4_omlp_only-kv_fp8_cast, and general/ptq/nvfp4_weight_only-kv_fp8_cast. These compose the same model-quant configs as their -kv_fp8 siblings with the kv_fp8_cast unit (constant-amax FP8 KV cache, no KV calibration forward pass).
• Add Nemotron-3-Super-120B-A12B PTQ recipes modelopt_recipes/models/Nemotron-3-Super-120B-A12B/super-nvfp4.yaml (MSE-mixed) and super-nvfp4-max-calib.yaml (max-calib mixed): NVFP4 W4A4 routed experts + FP8 per-tensor shared experts / Mamba in/out_proj + FP8 KV cache.
• Group layerwise calibration options under a nested LayerwiseConfig and add two knobs: get_qdq_activations_from_prev_layer (correct GPTQ-Hessian vs max-calib activation semantics — defaults to True for GPTQ, False for max/mse/local_hessian) and save_every (gate per-window next_inputs.pt activation-cache writes). Legacy bool layerwise and flat layerwise_checkpoint_dir keys still work; the bool form emits a DeprecationWarning.
• Add examples/alpamayo showing FP8, NVFP4, and AutoQuantize (mixed-precision) quantization of the Alpamayo (formerly Alpamayo-R1) ~10B vision-language-action model, with a joint VLM + diffusion calibration loop and both fake-quant and --real-quant packed-checkpoint export. See examples/alpamayo/README.md for details.
• Refactor llm_qat example with unified YAML-based configuration and flexible dataset blending. ModelOptArgParser adds --config YAML support with CLI overrides and auto-generates ARGUMENTS.md from dataclass definitions. Dataset blending (configs/dataset/blend.yaml) supports HuggingFace datasets, local JSON/JSONL/Parquet files, and weighted multi-source blends. The legacy FSDP1 accelerate config is removed; llm_qat now documents FSDP2, DeepSpeed, and DDP backends.

Megatron Framework (M-LM / M-Bridge)

• Add quantization examples for the Megatron-Bridge framework (examples/megatron_bridge/): post-training quantization (quantize.py calibrates an HF model via --quant_cfg alias / full config name or a --recipe YAML, with optional KV-cache quant, weight-only, compression, and MoE expert-ratio calibration, and saves a Megatron checkpoint with tensor / pipeline / expert parallelism), export to a deployable HuggingFace (unified) checkpoint for TensorRT-LLM / vLLM / SGLang (export.py), and Quantization Aware Distillation (extend existing distill.py). See examples/megatron_bridge/README.md for details.
• Add Megatron Core export/import mapping for Qwen3-VL (Qwen3VLForConditionalGeneration) vision-language models. The mapping handles the model.language_model. weight prefix used by Qwen3-VL.
• Add shared Megatron-Core calibration forward loop: modelopt.torch.utils.plugins.megatron_calibration.get_megatron_calibration_forward_loop produces the forward_loop callable expected by mtq.quantize / mtp.prune. Replaces the bespoke calibration loops in Megatron-LM and Megatron-Bridge for quantization and pruning with a single canonical implementation.
• Support Megatron-Core checkpoint restore and export for MSE NVFP4StaticQuantizer.
• Add mixed-precision FP8 + NVFP4 export for Megatron-Core: per-layer quant_algo recorded under quantized_layers in hf_quant_config.json, PP-aware kv_cache_dtype gather, fused-QKV exclude split into per-HF-name q/k/v_proj entries.
• Add support for active_params (for MoE models) and memory_mb constraints in Minitron pruning on top of existing params constraint. You can also provide multiple constraints. See examples/pruning/README.md for more details. The underlying utility functions mcore_param_count, mcore_memory_footprint_mb, and print_mcore_model_stats in modelopt.torch.nas.plugins.megatron_model_stats are also available for standalone use to compute parameter counts and memory footprints (weights + KV-cache + Mamba state) for any Megatron-Core model.
• Add Minitron pruning support for Megatron-Bridge Gemma3 models.
• Add end-to-end optimization tutorial for Minitron pruning + two-phase distillation (80B @ 8K + 20B @ 32K long-context = 100B tokens) + FP8 PTQ + vLLM deployment for Nemotron-3-Nano-30B-A3B-BF16 (MoE + Mamba-Transformer hybrid) → Pruned 22B/A3.0B active params, along with data blend preparation steps (with tool-calling data) and detailed pruning / data-blend / long-context ablations. See examples/megatron_bridge/tutorials/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16/README.md for details.

Datasets & Calibration

• Add DATASET_COMBOS to modelopt.torch.utils.dataset_utils — single --dataset tokens that fan out to multiple registered datasets; per-entry num_samples is split evenly across the members. Initial combos: cnn_nemotron_v2_mix (cnn_dailymail + nemotron-post-training-dataset-v2, used by hf_ptq.py when no --dataset is provided) and nemotron-post-training-v3 (the seven nvidia/Nemotron-* SFT datasets added in #1498, mirroring the [nemotron-post-training-v3 collection](https://h...

0.45.0rc3

Install the 0.45.0rc3 pre-release version using

pip install nvidia-modelopt==0.45.0rc3 --extra-index-url https://pypi.nvidia.com

0.45.0rc2

Install the 0.45.0rc2 pre-release version using

pip install nvidia-modelopt==0.45.0rc2 --extra-index-url https://pypi.nvidia.com

0.45.0rc1

Install the 0.45.0rc1 pre-release version using

pip install nvidia-modelopt==0.45.0rc1 --extra-index-url https://pypi.nvidia.com

0.45.0rc0

Install the 0.45.0rc0 pre-release version using

pip install nvidia-modelopt==0.45.0rc0 --extra-index-url https://pypi.nvidia.com

ModelOpt 0.44.0 Release

New Features

• Support full Transformer Engine spec for Minitron pruning (mcore_minitron). Now we no longer need to use custom ModelOpt spec. Note that this does not affect the usage of the pruning workflow but makes pruning slightly faster and may result in slightly different pruned model because of different kernel and numerics.
• Add end-to-end tutorial for Minitron pruning + distillation + quantization + evaluation + vLLM deployment for Nemotron-Nano-9B-v2 → Pruned 7B along with data blend preparation steps (and ablation study). See examples/pruning/minitron/README.md for details.
• Add Puzzletron - a new algorithm for heterogeneous pruning of LLM and VLM models. See examples/puzzletron/README.md for more details.
• Added iterator interface using CalibrationDataReader in ONNX quantization workflow.
• Add N:M sparse softmax support to the Triton flash attention kernel (modelopt.torch.kernels.common.attention.triton_fa). See examples/llm_sparsity/attention_sparsity/README.md for usage.
• Add skip-softmax skipping to the Triton flash attention kernel (modelopt.torch.kernels.common.attention.triton_fa). See examples/llm_sparsity/attention_sparsity/README.md for usage.
• Add Video Sparse Attention (VSA) method for video diffusion models (modelopt.torch.sparsity.attention_sparsity). VSA uses 3D block tiling with a two-branch architecture for attention speedup.
• Enable PTQ workflow for the Step3.5-Flash MoE model with NVFP4 W4A4 + FP8 KV cache quantization. See modelopt_recipes/models/Step3.5-Flash/nvfp4-mlp-only.yaml for more details.
• Add support for vLLM fakequant reload using ModelOpt state for HF models. See examples/vllm_serve/README.md for more details.
• [Early Testing] Add Claude Code PTQ skill (.claude/skills/ptq/) for agent-assisted post-training quantization. The skill guides the agent through environment detection, model support checking, format selection, and execution via the launcher or manual SLURM/Docker/bare GPU paths. Includes handling for unlisted models with custom module patching. This feature is in early testing — use with caution.
• [Early Testing] Polish Claude Code evaluation skill (.claude/skills/evaluation/) for agent-assisted LLM accuracy benchmarking via NeMo Evaluator Launcher. Adds two companion skills vendored verbatim from NVIDIA-NeMo/Evaluator: launching-evals (run/check/debug/analyze NEL evaluations) and accessing-mlflow (query MLflow runs, compare metrics, fetch artifacts). Re-sync at a pinned upstream SHA via .claude/scripts/sync-upstream-skills.sh. Also adds a shared skills/common/credentials.md covering HF / NGC / Docker token setup referenced by multiple skills. This feature is in early testing — use with caution.
• Add performant layerwise calibration for large models that don't fit on GPU (e.g. DeepSeek-R1, Kimi-K2). See modelopt_recipes/general/ptq/nvfp4_experts_only-kv_fp8_layerwise.yaml for usage. Layerwise calibration also supports PTQ with intermediate progress saving — useful when long PTQ runs get hit with Slurm timeouts. See modelopt_recipes/general/ptq/nvfp4_default-kv_none-gptq.yaml for usage.
• Add implicit GEMM CUDA kernel for Conv3D with fused NVFP4 fake quantization (modelopt.torch.quantization.src.conv). When NVFP4 quantization is applied to an nn.Conv3d layer via ModelOpt PTQ, the implicit GEMM path is used automatically instead of cuDNN. Uses BF16 WMMA tensor cores (SM80+) with FP32 accumulation and in-kernel FP4 (E2M1) activation quantization. Grouped convolution (groups > 1) falls back to the default cuDNN path. Inference only — training mode falls back to cuDNN with a warning.
• Add FP8 MHA quantization support for vision transformers. Adds an attention-aware ONNX post-processing pass (scale Mul / K-transpose move before Q, Q→DQ insertion on softmax output) in FP8QuantExporter (modelopt.onnx.export.fp8_exporter.FP8QuantExporter), per-instance nested-attention-wrapper skipping in the HF plugin, and nn.LayerNorm registration in QuantModuleRegistry so BMM input quantizers and LayerNorm output quantizers defined in FP8_DEFAULT_CFG are honored end-to-end. See examples/torch_onnx/torch_quant_to_onnx.py for the general timm-model quantize→ONNX workflow.

Backward Breaking Changes

• The quant_cfg field in quantization configs is now an ordered list of QuantizerCfgEntry dicts instead of a flat dictionary. Each entry specifies a quantizer_name wildcard, an optional parent_class filter, a cfg dict of quantizer attributes, and/or an enable flag. Entries are applied in list order with later entries overriding earlier ones. The old dict-based format is still accepted and automatically converted via normalize_quant_cfg_list(), but now emits a DeprecationWarning; new code should use the list format. All built-in configs (e.g. FP8_DEFAULT_CFG, INT4_AWQ_CFG, NVFP4_DEFAULT_CFG), examples, and YAML recipes have been updated. See the quant-cfg documentation for the new format reference and migration guide.
• Deprecated Mllama (Llama 3.2 Vision) support in the llm_ptq and vlm_ptq examples. The model_type == "mllama" branches and MllamaImageProcessor usage have been removed from hf_ptq.py and example_utils.py. For image-text calibration of VLMs, use --calib_with_images with a supported VLM (see Nemotron VL section in examples/llm_ptq/README.md).

Bug Fixes

• Fix Megatron utility functions for generation (with pipeline parallelism) and ~10x speedup in MMLU score evaluation (by batching prefill passes).
• Fix Minitron pruning (mcore_minitron) for MoE models. Importance estimation hooks were incorrectly registered for MoE modules and NAS step was hanging before this.
• Fix TRT support for remote autotuning in ONNX Autotune from 10.16+ to 10.15+ and fix TRT versioning check to the trtexec version instead of the TRT Python API when using trtexec backend.
• Exclude MatMul/Gemm nodes with K or N < 16 from ONNX INT8 and FP8 quantization. Such small-dimension GEMMs cannot efficiently use INT8/FP8 Tensor Cores and the added Q/DQ layers cause perf regressions in TensorRT. Honors Gemm transB when deriving K.
• Fix nvfp4_awq export AssertionError: Modules have different quantization formats for MoE models (e.g. Qwen3-30B-A3B) when some experts are not exercised by the calibration data. awq_lite now applies a neutral all-ones pre_quant_scale to any expert that ends up disabled (no cache-pass tokens, NaN scales, or no search-pass tokens) so its format remains nvfp4_awq, consistent with the rest of the MoE block. A warning is emitted whenever this fallback fires.

Misc

• [Security] Changed the default of weights_only to True in torch.load for secure checkpoint loading. If you need to load a checkpoint that requires unpickling arbitrary objects, first register the class in torch.serialization.add_safe_globals([cls]) before loading. Added safe_save (modelopt.torch.utils.serialization.safe_save) and safe_load (modelopt.torch.utils.serialization.safe_load) API to save and load checkpoints securely.
• Bump minimum required PyTorch version to 2.8.
• [Experimental] Add support for transformers>=5.0, including generic PTQ and unified HF checkpoint export for fused MoE expert modules (Mixtral, Qwen2-MoE, Qwen3-MoE, Qwen3.5-MoE, DeepSeek-V3, Jamba, OLMoE, etc.).
• Improve megatron_preprocess_data: add --reasoning_content support for Nemotron v3 datasets, eliminate intermediate JSONL for HuggingFace datasets, return output file prefixes from the Python API, add gzip input support (.jsonl.gz), add --strip_newlines flag for plain-text pretraining data, add --hf_streaming for very large datasets (only consumed rows downloaded), and auto-shuffle when --hf_max_samples_per_split is set to avoid biased sampling.
• Add installation support for Python 3.14. Only basic unit tests are verified for now. Production usage still defaults to Python 3.12. Python 3.10 support will be dropped in the next release.

0.44.0rc5

fix(te-plugin): handle TE 2.15+ tuple return from `_Linear` / `_Group…

0.44.0rc4

fix(te-plugin): make _Linear arg indexing robust to TE signature chan…

0.44.0rc3

0.44.0rc3

0.44.0rc2

Install the 0.44.0rc2 pre-release version using

pip install nvidia-modelopt==0.44.0rc2 --extra-index-url https://pypi.nvidia.com