ESMFold

Made the language-model-to-structure route practical at metagenomic-atlas scale, complementing MSA-centered folding systems. Its Science paper appeared in 2023 after a 2022 preprint; it is a structure predictor built on ESM-2, not another name for the language model.

Chapters

1. Read a sequence

   A frozen ESM-2 encoder produces contextual residue representations.

   Explain how sequence-only language features provide context for folding.

   • Layer Normalization
   • Rotary positions for queries and keys
   • Multi-head attention with masks and additive bias

2. Build folding representations

   Mix language-model layers, initialize single/pair features and exchange information through attention and triangle updates.

   Combine language-model layers into folding single and pair representations.

   • Weighted Layer Mixture
   • ESMFold Single-to-Pair Features
   • Triangle multiplicative update: incoming and outgoing

3. Construct and recycle atoms

   Residue frames and torsions produce atoms; previous state and geometry enter later trunk passes.

   Move from residue frames to atoms and describe recycling between trunk passes.

   • Batched rigid transforms
   • Local Quaternion Frame Update
   • ESMFold Recycling Update

4. Report the prediction

   Learned confidence and distogram heads accompany the predicted coordinates.

   Read learned confidence summaries alongside a predicted structure.

   • Linear-Stack pLDDT Head
   • pLDDT Confidence Summary
   • Predicted TM Score

All exercises in this guide

• Masked Sum
• Masked Mean
• Layer Normalization
• Exact GELU
• Affine Projection
• Feedforward Block
• Masked Softmax
• Multi-head attention with masks and additive bias
• Rotary positions for queries and keys
• LayerNorm and RMSNorm from reductions
• Weighted Layer Mixture
• Token Embedding Lookup
• ESMFold Relative Position Buckets
• ReLU
• Activations and gated feed-forward transitions
• Learned language-model layer mixtures
• Chain, entity and token-relative position buckets
• Affine projection and embedding lookup
• ESMFold Single-to-Pair Features
• Atom Layout Gather
• Atom–Token Broadcast and Pool
• Single-to-pair features and atom reference conditioning
• Triangle multiplicative update: incoming and outgoing
• Triangle attention: starting and ending node
• Stable vector norms, normalization, dot and cross features
• Batched rigid transforms
• Masked pairwise distances and hard distance-bin targets
• Invariant point attention (IPA)
• Local Quaternion Frame Update
• Normalized-quaternion backbone updates
• Learned torsions and hierarchical atom reconstruction
• ESMFold Recycling Update
• Recycling representations and clean-template state
• Linear-Stack pLDDT Head
• Distogram Linear Head
• PAE Linear Head
• pLDDT Confidence Summary
• Predicted TM Score
• Learned structural confidence heads and model-aware ranking
• ESM-2 token-dropout scaling
• Local distance difference test (lDDT)
• Geometric confidence targets and observation masks