ESM-3

A landmark in multimodal protein generation: sequence, structural tokens, and functional annotations are modeled together rather than attaching a structure predictor to a sequence-only language model. The Science paper reports experimentally fluorescent generated proteins. Its evolutionary-distance comparison is an estimate, not a literal simulation of evolution, and those results should not be attributed automatically to the smaller open checkpoint.

Chapters

1. Prepare aligned tracks

   Local geometry is encoded and quantized into structure tokens, then combined with sequence and annotation embeddings.

   Convert geometry into structure tokens and combine aligned input tracks.

   • Right-handed local frame from three points
   • Nearest structural-code quantization
   • ESM3 multitrack embedding and confidence fusion

2. Reason across residues

   Rotary sequence attention and SwiGLU transitions form the main stack; the first open-model block also uses geometric attention.

   Distinguish sequence attention, geometric reasoning and channel updates.

   • Rotary positions for queries and keys
   • Geometric direction-and-distance attention
   • Gated Feedforward Block

3. Predict and reveal tokens

   Track-specific heads support masked supervision and iterative completion of unknown positions.

   Predict missing track tokens and reveal a bounded subset of them.

   • Categorical Cross-Entropy
   • Constrained categorical sampling with tied positions
   • Iterative masked-token reveal with fixed prompts

4. Decode the result

   A separate structural decoder produces backbone geometry and confidence from structure tokens.

   Explain how structure tokens feed a separate geometric decoder.

   • ESM3 Single-to-Pair Features
   • GELU pLDDT Head
   • pLDDT Confidence Summary

All exercises in this guide

• SiLU
• Affine Projection
• Feedforward Block
• Gated Feedforward Block
• Masked Sum
• Masked Mean
• Stable vector norms, normalization, dot and cross features
• Masked pairwise distances and hard distance-bin targets
• Masked batched kNN graphs and neighbor gather
• Batched rigid transforms
• Right-handed local frame from three points
• Chain, entity and token-relative position buckets
• Masked Softmax
• Geometric direction-and-distance attention
• Affine projection and embedding lookup
• Nearest structural-code quantization
• Ragged segment sum and mean
• Categorical and numeric molecular feature encoding
• ESM3 multitrack embedding and confidence fusion
• Layer Normalization
• Multi-head attention with masks and additive bias
• Rotary positions for queries and keys
• LayerNorm and RMSNorm from reductions
• Exact GELU
• Activations and gated feed-forward transitions
• Constrained categorical sampling with tied positions
• Iterative masked-token reveal with fixed prompts
• ESM3 Single-to-Pair Features
• Atom Layout Gather
• Atom–Token Broadcast and Pool
• Single-to-pair features and atom reference conditioning
• GELU pLDDT Head
• PAE GELU Head
• pLDDT Confidence Summary
• Predicted TM Score
• PDE GELU Head
• Learned structural confidence heads and model-aware ranking
• Masked Log-Sum-Exp
• Categorical Cross-Entropy
• Binary Cross-Entropy with Logits
• Stable categorical and binary classification losses