Foundation Models Meet Single-Cell RNA-seq: A New Paradigm
The analogy between language models and single-cell genomics models is imperfect in several ways, but it is useful for one specific reason: both domains are defined by learning structure from enormous quantities of data that would be uninterpretable by direct inspection. In language modelling, the units are tokens. In single-cell genomics, the units are cells, each described by a sparse high-dimensional vector of gene expression measurements. In both cases, the insight is that the structure latent in the data, learned from scale, turns out to be far more transferable across tasks than anyone expected when the training began.
That transferability is the core reason foundation models matter for drug target discovery, not just for single-cell biology as a computational exercise.
What a Foundation Model Learns from Cell Profiles
A single-cell RNA-seq profile is a measurement of how actively each gene is being transcribed in a single cell at the moment of capture. In practice, each profile measures expression across roughly 20,000 protein-coding genes, with the majority of counts being zero due to both biological reality and technical dropout. The resulting data matrix for a large study might contain 500,000 cells by 20,000 genes, mostly sparse.
Training a transformer-class model on this kind of data requires representing each cell in a way the architecture can process. The most common approach treats non-zero gene expression values as tokens, ordering them by expression rank or magnitude. The model learns, through masked gene prediction tasks analogous to masked language modelling, to predict missing expression values given the rest of a cell's profile. Through this process, it builds internal representations that encode cell identity, cell state, and gene co-expression structure implicitly, without any explicit cell-type labels during pretraining.
The resulting cell embeddings capture biological similarity in a compact form. Cells of the same type cluster in embedding space even across experiments with different technical protocols, different donors, and different tissue preparation methods. This cross-study coherence is what makes the representations useful as a foundation for downstream tasks: you are not fitting a model to one study's idiosyncrasies; you are working with a representation that generalises across the biology.
Scale and What It Buys You
The Human Cell Atlas has catalogued over 50 million cells across more than 33 organs. Repositories like CELLxGENE aggregate millions more from public studies. Training a foundation model on data at this scale, from diverse tissues, diseases, donors, and protocols, produces representations with properties that smaller-scale models cannot match.
The most important property for our purposes is rare cell-type representation. A disease-relevant cell subpopulation might constitute 2-3% of cells in an inflamed tissue and appear in perhaps 15-20 studies in the public literature. At training scale, that still represents tens of thousands of individual cell profiles. The model develops meaningful representations for those cells: it knows their co-expression programmes, their relationship to neighbouring cell states, and how their profile deviates from resting counterparts. A model trained on a single study or tissue type cannot represent them reliably.
The second property is interpretable gene-level structure. Because the training objective involves predicting gene expression from context, the model's attention mechanism learns to identify which genes are informative for predicting the expression of others. In practice, this means the model encodes something like gene regulatory programme structure. When you query the model about a target gene's role in a specific cell type, the attention patterns are biologically meaningful, not arbitrary.
From Pretraining to Target Discovery
The pretraining task produces a general-purpose cell encoder. That encoder is the starting point, not the end product, for drug target discovery applications.
The downstream application works by querying the model with a disease state: a set of cells from diseased tissue profiled at single-cell resolution, compared to matched healthy controls. The model encodes both populations, and the comparison reveals which cell types show the largest shifts in expression programme, what the direction of those shifts is (activated, transitional, exhausted, pathologically polarised), and which gene programmes are differentially active across those shifts.
Target genes emerge from this analysis as genes that are differentially expressed in the disease-associated cell state, with high expression specificity to that cell type relative to others in the tissue. The foundation model adds value over a conventional differential expression analysis in two specific ways. First, it can identify cell types that are defined by programme shifts rather than marker gene presence, including transitional states that would not have clean cluster boundaries in a standard UMAP. Second, it provides a context for evaluating each candidate gene: how that gene behaves across the full range of cell types the model has seen, rather than just in the current dataset.
Where the Analogy to Language Models Breaks Down
The language model analogy is instructive but has real limits. Text tokens have discrete identity: "cat" is "cat." A gene's expression value is continuous, it is confounded by technical factors, and its relationship to cell biology is mediated by a regulatory layer that language models do not have to worry about. You cannot simply scale up a text transformer and expect it to model gene regulation correctly.
The practical consequence is that architectural choices matter significantly. How you encode expression magnitudes, whether you condition on cell cycle state, how you handle the extremely sparse protein-coding subsets that dominate real cell profiles, and how you normalise across library sizes: each of these decisions affects whether the learned representations track real biology or technical covariation.
Similarly, evaluating a cell foundation model requires domain-specific benchmarks that do not have direct equivalents in NLP. Annotation transfer accuracy on held-out datasets, cross-study cell-type clustering consistency, perturbation response prediction against CRISPR screens, and gene programme coherence in embedding space are the tests that matter. Benchmark performance on cell classification tasks alone is not sufficient to assess whether the model is learning the biology needed for target discovery.
What This Means for Discovery Teams in Practice
The practical implication for a discovery team is a shift in what the computational front end of target selection can deliver. Instead of running a differential expression analysis on available bulk data and getting a list of genes to investigate further in the lab, you can query a trained model with your disease context and get back a ranked set of cell-type-specific target hypotheses, each with an expression specificity profile across human tissue cell types.
That does not replace the need for functional validation. A gene that appears as a strong candidate in the model output still needs cell line work, target engagement studies, and eventually in vivo evidence. The model does not validate targets; it prioritises the order in which your team investigates them, and it provides a biological rationale grounded in human single-cell data rather than in mouse models or transformed cell lines that may or may not reflect the relevant human biology.
The bottleneck the model addresses is the one between "we have a disease indication" and "we have three or four mechanistically coherent target hypotheses worth committing resources to." That bottleneck has historically been expensive, slow, and often resolved by the gut feel of the most experienced biologist in the room. A well-trained foundation model makes that process more systematic, more anchored in human data, and considerably faster.
