ML-KEM assembly: AVX512F and AVX512VBMI

[SparkiDev]

Description

New assembly for ML-KEM for Intel x64 machines that support AVX512 extensions.

Testing

Regression testing of ML-KEM.

Generator

https://github.com/wolfSSL/scripts/pull/638

[github-actions]

MemBrowse Memory Report

linuxkm-pie

• Data: __patchable_function_entries +64 B (+0.2%, 26,320 B)

linuxkm-standard

• Data: __patchable_function_entries +8 B (+0.0%, 48,984 B)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10981

Scan targets checked: wolfcrypt-bugs, wolfcrypt-rs-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src

No new issues found in the changed files. ✅

[Frauschi]

🐺 Skoll Code Review

Overall recommendation: APPROVE
Findings: 1 total — 1 posted, 0 skipped

Posted findings

• [Medium] New AVX512 k2 noise path uses large unconditional stack buffers, unlike its AVX2 sibling — wolfcrypt/src/wc_mlkem_poly.c:4754-4809

Review generated by Skoll via Claude/Codex

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10981

Scan targets checked: wolfcrypt-bugs, wolfcrypt-rs-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src

No new issues found in the changed files. ✅

[Frauschi]

🐺 Skoll Code Review

Overall recommendation: APPROVE
Findings: 18 total — 5 posted, 13 skipped

Posted findings

• [Medium] mlkem_get_noise_k3/k4_avx512 double the stack buffer and drop the WOLFSSL_SMALL_STACK handling used by the k2 sibling — wolfcrypt/src/wc_mlkem_poly.c:4877
• [Medium] NO_AVX512_SUPPORT no longer disables all AVX-512 code emission: the VBMI assembly block is not nested under the AVX-512 guard — wolfssl/wolfcrypt/wc_mlkem.h:586-598
• [Low] New AVX512 declarations are inconsistently const-qualified, named, and ordered — wolfssl/wolfcrypt/wc_mlkem.h:696-726
• [Low] Doc comments on the new x8 helpers omit @return and misstate lane counts — wolfcrypt/src/wc_mlkem_poly.c:4581-4587
• [Low] NO_AVX512_VBMI_SUPPORT does not remove all VBMI-family instructions: vpcompressw is emitted from the plain AVX-512 block — wolfssl/wolfcrypt/wc_mlkem.h:591-594

Skipped findings

• [Medium] AVX512BW-dependent ML-KEM assembly is dispatched on the AVX512F CPUID bit alone
• [Medium] Four dispatch tiers added with no test that deterministically exercises each one
• [Medium] ML-KEM AVX-512 dispatch gated on the AVX-512F CPUID bit while the assembly requires AVX-512BW (and AVX-512VL)
• [Low] mlkem_csubq_avx512 is declared but never called and has no AVX2 counterpart declaration
• [Low] New _ins suffix is undocumented and four helpers are generated by a function-defining macro
• [Low] VBMI guard nesting in wc_mlkem.h does not actually match the assembly guard it claims to mirror
• [Low] mlkem_get_noise_k3_avx512 / k4_avx512 use 1 KB plain stack arrays, ignoring WOLFSSL_SMALL_STACK
• [Low] New AVX-512 CPUID flags are set from feature bits with no OSXSAVE/XCR0 state check
• [Info] AVX-512 matrix generators roughly double the non-small-stack frame; k2 pays the full eight-lane cost to use four lanes
• [Info] The four-lane mlkem_redistribute_*_rand_avx512 dispatch is unreachable in normal builds
• [Info] mlkem_csubq_avx512 is newly declared in the header but has no caller
• [Info] New x8 noise helpers can return MEMORY_E but their doc comments omit @return; k3 comment miscounts lanes
• [Info] No test exercises the new CPUID dispatch variants, so cross-implementation output equivalence is unverified

Review generated by Skoll via Claude/Codex

[SparkiDev]

Jenkins: retest this please

[SparkiDev]

Jenkins: retest this please

Visual Studio: CRL monitor test failed.

[Frauschi]

🐺 Skoll Code Review

Overall recommendation: REQUEST_CHANGES
Findings: 15 total — 1 posted, 14 skipped

Posted findings

• [High] AVX512-VBMI/VBMI2 dispatch branches never test CPUID_AVX512, so cpuid_clear_flag(CPUID_AVX512) no longer disables AVX512 code — wolfcrypt/src/wc_mlkem_poly.c:5799-5812

Skipped findings

• [Medium] New WOLFSSL_MLKEM_ASM_TEST macro gates assembly with no prototypes and no in-tree consumer
• [Medium] No test exercises the four new rejection-sampler variants or the AVX512 paths on non-AVX512 CI
• [Medium] mlkem_get_noise_k2_avx512 does not advance seed[WC_ML_KEM_SYM_SZ] while its AVX2 sibling does
• [Medium] AVX512 matrix generators double the stack frame in default (non-small-stack) builds
• [Medium] No configure-time probe for the new NO_AVX512_VBMI_SUPPORT / NO_AVX512_VBMI2_SUPPORT opt-outs
• [Medium] AVX-512 VBMI/VBMI2 dispatch bypasses the CPUID_AVX512 master flag, defeating cpuid_clear_flag(CPUID_AVX512)
• [Medium] New AVX-512 matrix generators roughly double single-frame stack usage in non-WOLFSSL_SMALL_STACK builds
• [Low] mlkem_csubq_avx512 is declared and assembled but never called
• [Low] New CPUID flag defines break column alignment and are missing from the benchmark feature banner
• [Low] Unused lanes 4..7 in mlkem_gen_matrix_k2_avx512 duplicate lane 0's XOF input and omit the (word32) cast
• [Low] The _ins suffix on the new dispatch helpers is undocumented and novel to the codebase
• [Low] New static dispatch wrappers are unreferenced in single-parameter-set builds
• [Info] SGX CPUID path cannot express the new VBMI/VBMI2 flags
• [Info] AVX-512 and AVX-2 ML-KEM-512 noise paths now differ in whether they mutate seed[WC_ML_KEM_SYM_SZ]

Review generated by Skoll via Claude/Codex

[Frauschi]

🟠 [High] AVX512-VBMI/VBMI2 dispatch branches never test CPUID_AVX512, so cpuid_clear_flag(CPUID_AVX512) no longer disables AVX512 code
🚫 BLOCK bug

Every new VBMI/VBMI2 dispatch branch added by this diff gates only on the sub-feature flag and never on IS_INTEL_AVX512(cpuid_flags). The affected sites are mlkem_vec_compress_10 (~5799), mlkem_vec_decompress_10 (~6024), mlkem_vec_decompress_11 (~6126), mlkem_compress_4 (~6309), mlkem_compress_5 (~6401), mlkem_from_bytes (~6858), mlkem_to_bytes (~6951), plus mlkem_rej_uniform_n_ins/mlkem_rej_uniform_ins (2326-2369). Because the VBMI branch is tested before the AVX512 branch, it wins whenever the VBMI CPUID bit is set.

wolfSSL exposes cpuid_clear_flag() as a WOLFSSL_API (declared wolfssl/wolfcrypt/cpuid.h:197, implemented wolfcrypt/src/cpuid.c:710) precisely so integrators can switch an instruction set off at run time. After this diff, cpuid_clear_flag(CPUID_AVX512) leaves CPUID_AVX512_VBMI/CPUID_AVX512_VBMI2 set, and ML-KEM keeps executing AVX512 instructions through the *_vbmi/*_vbmi2 routines. The AVX512-only branch immediately below is correctly gated, so the behaviour is inconsistent within the same function.

This also diverges from the established repo convention. wolfcrypt/src/wc_frodokem_mat.c:1723-1735 nests the sub-feature test (IS_INTEL_VAES) inside a block already gated on IS_INTEL_AVX512(cpuid_flags) && (SAVE_VECTOR_REGISTERS2() == 0). The ML-KEM code inverts that structure.

(The narrow architectural point that VBMI implies AVX512F on real silicon does not save this: the flags are software state that callers are allowed to modify, and the code depends on an unstated invariant.)

Suggestion:

Suggested change

	#ifdef WOLFSSL_MLKEM_HAVE_INTEL_AVX512_VBMI
	#ifdef WOLFSSL_MLKEM_HAVE_INTEL_AVX512_VBMI
	if (IS_INTEL_AVX512(cpuid_flags) && IS_INTEL_AVX512_VBMI(cpuid_flags) &&
	(SAVE_VECTOR_REGISTERS2() == 0)) {
	mlkem_compress_10_avx512_vbmi(r, v, (int)k);
	RESTORE_VECTOR_REGISTERS();
	}
	else
	#endif

[SparkiDev]

fixed