Geneformer

An early landmark connecting large-corpus rank-token pretraining to small-data network-biology tasks, with targeted cardiomyocyte experimental follow-up in its original study. It is featured for the transfer-learning idea and scientific hypotheses, not for proven universal cell simulation or best zero-shot performance.

Chapters

1. Rank the transcriptome

   Normalize expression with supplied corpus medians, then embed gene identities and rank positions.

   Rank normalized expression and embed gene identity separately from rank.

   • Corpus-normalized gene rank tokenization
   • Token Embedding Lookup

2. Encode gene context

   A bidirectional BERT-style encoder produces representations and supports masked gene-identity supervision.

   Contextualize ranked genes and supervise selected gene identities.

   • Multi-head attention with masks and additive bias
   • Feedforward Block
   • Categorical Cross-Entropy

3. Read and adapt representations

   Pool contextual states or train a task-specific gene/cell classifier.

   Pool valid contextual states or apply a task-specific readout.

   • Masked Mean
   • Categorical Cross-Entropy

4. Edit, re-encode and compare

   Delete or promote gene tokens, run the encoder again, then compare the resulting states.

   Edit gene tokens, re-encode the sequence and compare contextual representations.

   • Delete Gene Tokens
   • Promote Gene Tokens
   • Paired gene-embedding perturbation statistics

All exercises in this guide

• Masked Sum
• Masked Mean
• Corpus-normalized gene rank tokenization
• Token Embedding Lookup
• Layer Normalization
• ReLU
• Affine Projection
• Affine projection and embedding lookup
• Feedforward Block
• Masked Softmax
• Multi-head attention with masks and additive bias
• LayerNorm and RMSNorm from reductions
• Activations and gated feed-forward transitions
• Masked sums, means and weighted reductions
• Delete Gene Tokens
• Promote Gene Tokens
• Gene-token deletion and rank promotion
• Stable vector norms, normalization, dot and cross features
• Paired gene-embedding perturbation statistics
• Masked Log-Sum-Exp
• Categorical Cross-Entropy
• Stable categorical and binary classification losses