MODEL LEARNING PATH
ESM-2
ESM-2 reads a protein sequence in both directions. A stack of rotary attention and GELU transitions produces residue states, masked-token predictions and optional contact scores.
Loading the interactive page…
A landmark in scaling masked protein language models and reusing their learned representations. Later work initialized DPLM from ESM-2 and used ESM-2 in an experimentally tested biofoundry evolution workflow. This is evidence of conceptual influence and downstream reuse, not a claim that ESM-2 was the first protein language model or independently achieved those pipelines' results.
Chapters
Embed the sequence
Token lookup and mask-token rescaling prepare the encoder input.
Turn residue identities into embeddings and apply mask-token rescaling.
Build contextual states
Bidirectional rotary attention and GELU transitions repeat, followed by final LayerNorm.
Compose rotary attention, an ungated GELU transition and normalization.
Read predictions
A tied vocabulary head predicts residue identity; a separate attention-based head predicts contacts.
Separate residue logits from attention-derived contact probabilities.
Train with masked targets
Only selected target identities contribute to the masked-token objective.
Compute identity loss only at selected masked positions.
All exercises in this guide
- Token Embedding Lookup
- Affine Projection
- Affine projection and embedding lookup
- Masked Sum
- Masked Mean
- ESM-2 token-dropout scaling
- Rotary positions for queries and keys
- Masked Softmax
- Multi-head attention with masks and additive bias
- Exact GELU
- Feedforward Block
- Layer Normalization
- Activations and gated feed-forward transitions
- LayerNorm and RMSNorm from reductions
- Stable categorical and binary classification losses
- Tied vocabulary head and masked/causal token losses
- Attention-derived protein contact prediction
- Stable logsumexp, log-softmax and masked softmax