Evo

A distinct generative-genomics landmark: a single DNA language-modeling objective supported prediction and generation across protein, RNA, and DNA systems, with experiments on selected molecular designs. This page covers Evo 1 (Science 2024), not the different Evo 2 successor.

Chapters

1. Embed and normalize tokens

   Character-level DNA tokens initialize a residual stream and enter the selected block type.

   Embed character tokens and normalize their feature vectors.

   • Token Embedding Lookup
   • RMS Normalization

2. Mix sequence information

   Hyena uses a short filter, a modal long filter and multiplicative gates; interleaved attention blocks provide an alternative mixer.

   Trace causal filtering, modal recurrence and multiplicative mixing.

   • Causal depthwise filtering with a FIR cache
   • Real-imaginary modal filters and recurrent state
   • Complete Evo-style Hyena gated block

3. Update channels and predict

   A gated channel MLP refines each block output; the final readout produces next-token logits.

   Map mixed sequence features through a channel MLP to next-token logits.

   • Gated Feedforward Block
   • Affine Projection
   • Categorical Cross-Entropy

4. Generate with state

   Sampling chooses a token and updates the appropriate attention, short-filter and modal caches.

   Advance generation while retaining the appropriate mixer state.

   • Constrained categorical sampling with tied positions
   • Autoregressive KV-cache append and causal attention

All exercises in this guide

• Token Embedding Lookup
• Masked Sum
• Masked Mean
• Layer Normalization
• RMS Normalization
• LayerNorm and RMSNorm from reductions
• Dense 1-D dilated convolution
• Causal depthwise filtering with a FIR cache
• Causal long convolution with FFT
• Real-imaginary modal filters and recurrent state
• Affine Projection
• Feedforward Block
• Complete Evo-style Hyena gated block
• Masked Softmax
• Multi-head attention with masks and additive bias
• Rotary positions for queries and keys
• Exact GELU
• Gated Feedforward Block
• Activations and gated feed-forward transitions
• Affine projection and embedding lookup
• Constrained categorical sampling with tied positions
• Autoregressive KV-cache append and causal attention
• Masked Log-Sum-Exp
• Categorical Cross-Entropy
• Stable categorical and binary classification losses