Codon-level information complements protein language models for missense-variant effect prediction

We compare codon-level language-model scores from CaLM with residue-level protein language-model scores from ESM-2 and ESM-1b to test whether codon-resolution information contributes to missense-variant interpretation beyond protein-level constraints.

Overview

The main analyses use ClinVar and ClinMAVE variant datasets to evaluate:

• whether CaLM contributes signal beyond PLM baselines and PLM-PLM ensembles
• whether the CaLM contribution persists after explicit mutational-context controls
• how modality contribution varies across functional classes in ClinMAVE
• whether codon-level discordance remains after aggregating CaLM probabilities into amino-acid space
• whether gene-level CaLM contribution varies across genes and PLM backgrounds
• whether inferred modality contribution differs across matched DMS and CBGE assay contexts

Repository Structure

analysis/               Statistical analyses and model-scoring code
  scoring/              Data preparation and CaLM/PLM scoring utilities
plot/
  main/                 Main-figure scripts
  supplementary/        Supplementary-figure scripts for Figs. S1--S8
Figure/                 Final manuscript figures
Results/                Final result tables

Main Analyses

ClinVar model and context analyses:

• analysis/clinvar_ensemble.py: PLM/CaLM model-combination analysis for Fig. 2A--B
• analysis/clinvar_context.py: mutational-context control for Fig. 2C--D
• analysis/clinvar_discordance.py: amino-acid aggregation and substitution-discordance analysis for Fig. 5
• analysis/clinvar_gene_prepare.py: preparation of gene-level analysis tables
• analysis/clinvar_gene.py: gene-level codon-contribution analysis for Fig. 6

ClinMAVE analyses:

• analysis/clinmave_classes_esm2.py: functional-class analysis using ESM-2 (650M)
• analysis/clinmave_classes_esm1b.py: functional-class analysis using ESM-1b (650M)
• analysis/clinmave_pairs_esm2_150m.py: matched-pair analysis using ESM-2 (150M)
• analysis/clinmave_pairs_esm2_650m.py: matched-pair analysis using ESM-2 (650M)
• analysis/standardize_figure_weights.py: fold-specific scale adjustment for reported ensemble weights
• analysis/clinmave_context_inference.py: gene-block inference for matched DMS--CBGE weight shifts

Dependencies

The analysis scripts are Python-based and use common scientific packages, including:

• numpy
• pandas
• scipy
• scikit-learn
• statsmodels
• matplotlib
• seaborn
• torch
• fair-esm

CaLM should be installed from the original CaLM repository:

pip install torch torchvision torchaudio
pip install einops rotary_embedding_torch biopython
python setup.py install

ESM models can be installed using:

pip install fair-esm

Data Sources

Variant annotations were obtained from ClinVar and ClinMAVE:

• ClinVar: https://ftp.ncbi.nlm.nih.gov/pub/clinvar
• ClinMAVE: https://ngdc.cncb.ac.cn/clinmave/

Language-model resources:

• CaLM: https://github.com/oxpig/CaLM
• ESM: https://github.com/facebookresearch/esm

References

1. Rives, Alexander, et al. "Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences." Proceedings of the national academy of sciences 118.15 (2021): e2016239118.
2. Lin, Zeming, et al. "Evolutionary-scale prediction of atomic-level protein structure with a language model." Science 379.6637 (2023): 1123-1130.
3. Outeiral, Carlos, and Charlotte M. Deane. "Codon language embeddings provide strong signals for use in protein engineering." Nature Machine Intelligence 6.2 (2024): 170-179.
4. Meier, Joshua, et al. "Language models enable zero-shot prediction of the effects of mutations on protein function." Advances in neural information processing systems 34 (2021): 29287-29303.