Evaluation pitfalls for virtual-cell models: a checklist drawn from the benchmark and critique records

Research question

• Which evaluation pitfalls do the benchmark and critique papers in the library identify?
• For each pitfall, which record is the source and how mature is that source (peer reviewed or preprint)?

Current understanding

Our reading of the library records: the benchmark and critique papers point to a recurring set of evaluation issues: simple or control-informed baselines that are hard to beat, response shared across perturbations, miscalibrated metrics, zero-shot evaluation of embeddings, interpretability scores that add nothing beyond baselines, and scale that does not by itself add predictive diversity. Several of the proposed remedies are recorded only as preprints, and one peer-reviewed paper takes a counter-position on whether deep models beat baselines. This page is a checklist from the records, not a verdict. Certainty: high that the papers say this; the issues interact and are not ranked here.

Scope and how to read this page. Scope: library records only (title, venue, one-line summary and verification source); we have not read the full texts for this page, and every statement below is limited to what those records state. Preprints are marked and are not peer reviewed. This page makes no claim about glioma or other tumours. It is a checklist, not a ranking, and it does not repeat the detailed comparison on the linear-baselines page. VC0048 VC0052

1. Baselines. The records state that simple baselines are strong: deep-learning gene perturbation effect prediction does not yet outperform simple linear baselines (VC0048); a CRISPR-informed mean (the training-set perturbed-cell mean with the target gene set from prior knowledge) is reported to beat fine-tuned scGPT, GEARS and simple neural baselines on three datasets, and the paper also provides a corrected Adamson dataset (VC0144); a benchmark of 11 datasets reports it is uncommon for expression-forecasting methods to beat simple baselines (VC0145); a benchmark of foundation cell models against simple baselines is also recorded (VC0049). Certainty: high that the papers report this; what counts as the baseline differs between papers. VC0048 VC0144 VC0145 VC0049

2. Shared response across perturbations. Systema is recorded as a framework for evaluating perturbation prediction beyond systematic variation (Nature Biotechnology 2025). Our paraphrase of the concern: a model can look good if most perturbations move cells in a similar direction, and datasets may differ in how much of this they contain. Certainty: high for the framework as recorded; magnitudes per dataset are in the paper, which we cite but do not repeat. VC0052

3. Metric calibration and principles. A Nature Biotechnology paper (authors from Shift Bioscience, with a competing-interest statement) argues that MSE and Pearson delta are often miscalibrated and that deep models can beat uninformative baselines on calibrated metrics, over 14 datasets and 18 metrics (VC0146). A preprint proposes principles for evaluating virtual-cell perturbation models around signal, bounds and baselines (VC0053), and another preprint reports that the baseline gap persists in an empirical comparison of virtual-cell models (VC0054). A further preprint benchmarks foundation-model representations multimodally for cellular perturbation response (VC0055). Certainty: VC0146 peer reviewed with a disclosed competing interest; VC0053, VC0054 and VC0055 are preprints, not peer reviewed. VC0146 VC0053 · preprint VC0054 · preprint VC0055 · preprint

4. Zero-shot evaluation of embeddings and perturbation tasks. A Genome Biology paper reports zero-shot limitations of scGPT and Geneformer (VC0050), and a bioRxiv preprint benchmarks single-cell foundation models for perturbation effect prediction (VC0051; preprint). VC0050 VC0051 · preprint

5. Interpretability and scale. A BMC Genomics study reports that attention-derived edge scores in single-cell foundation models add no gene-regulatory-network information beyond baselines (VC0056), and a PLoS Computational Biology paper assesses scale and predictive diversity of Geneformer-based models (VC0057). Certainty: high that the papers are as recorded; the library holds only these studies on these questions. VC0056 VC0057

Evidence map

Primary studies & datasets

• VC0048 Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines. — Ahlmann-Eltze C, Huber W, Anders S., Nature methods, 2025
• VC0052 Systema: a framework for evaluating genetic perturbation response prediction beyond systematic variation. — Viñas Torné R, Wiatrak M, Piran Z, Fan S, Jiang L, Teichmann SA, Nitzan M, Brbić M., Nature biotechnology, 2025
• VC0144 Simple controls exceed best deep learning algorithms and reveal foundation model effectiveness for predicting genetic perturbations. — Wong DR, Hill AS, Moccia R., Bioinformatics, 2025
• VC0145 A comparison of computational methods for expression forecasting. — Kernfeld E, Yang Y, Weinstock JS, Battle A, Cahan P., Genome biology, 2025
• VC0146 Deep learning perturbation models can outperform baselines on calibrated metrics. — Miller HE, Mejia GM, Leblanc FJA, Swain B, Wang B, de Lima Camillo LP., Nature biotechnology, 2026

Related models & tools

• VC0053 · preprint Signal, Bounds, and Baselines: Principles for Evaluating Virtual Cell Perturbation Models — Vollenweider M, Bühlmann P., bioRxiv, 2026
• VC0054 · preprint An Empirical Comparison of Virtual Cell Models: Perturbation Prediction, Representation, and the Baseline Gap — Denvis O., Research Square, 2026
• VC0055 · preprint Multimodal Benchmarking of Foundation Model Representations for Cellular Perturbation Response Prediction — Hasanaj E, Cole E, Mohammadi S, Addagudi S, Zhang X, Song L, Xing EP., bioRxiv, 2025
• VC0051 · preprint PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction — Wenteler A, Occhetta M, Branson N, Curean V, Huebner M, Dee W, Connell W, Chung SP, Hawkins-Hooker A, Ektefaie Y, Valdez Córdova CM, Gallagher-Syed A., bioRxiv, 2024

Perturbation & validation evidence

• VC0049 Benchmarking foundation cell models for post-perturbation RNA-seq prediction. — Csendes G, Sanz G, Szalay KZ, Szalai B., BMC genomics, 2025
• VC0050 Zero-shot evaluation reveals limitations of single-cell foundation models. — Kedzierska KZ, Crawford L, Amini AP, Lu AX., Genome biology, 2025
• VC0056 Systematic evaluation of single-cell foundation model interpretability: attention-derived edge scores add no incremental value over gene-level features for perturbation-target prediction. — Kendiukhov I., BMC genomics, 2026
• VC0057 Assessing scale and predictive diversity in models for single-cell transcriptomics based on Geneformer. — Chen J, Schmidt F, Henao R., PLoS computational biology, 2026

13 library items cited on this page; 4 are preprints (not peer reviewed).

Related reviews

• Glioblastoma cell states and AI virtual cells
• GBM cell-state atlases and the virtual-cell foundation (draft, pending review)

Entities on this page

• Single-cell foundation model Methods & concepts
• GEARS Models
• Geneformer Models
• Simple linear baseline Methods & concepts
• Perturbation response prediction Methods & concepts
• scGPT Models
• Systematic variation Methods & concepts
• Zero-shot evaluation Methods & concepts

What remains uncertain

The pitfalls are drawn from record-level summaries, and full-text checks of every claim have not been done for this page; where the linear-baselines page reports our full-text reading, that page is the more detailed source. VC0048 VC0146

Whether deep models outperform baselines remains contested in the records (VC0048, VC0145 against VC0146), and the studies differ in datasets, baselines and metrics, so no single pitfall explains the disagreement. VC0048 VC0145 VC0146

Related questions

• When papers report that a mean or linear baseline matches deep models, are they comparing the same object, with the same metric, on the same kind of perturbations?
• Which mechanisms could make a perturbation-agnostic predictor look good (shared response, noise, target gene, metric), and which papers actually test which?
• How are virtual cell models validated?
• Do deep learning models beat linear baselines for perturbation prediction?
• What do the cell-eval metrics used by the Challenge actually reward, when checked against code and public data?
• Where do the two main baseline critiques (VC0048, VC0052) agree, conflict, or rest on unstated assumptions?
• Which single-cell foundation models are in the library, and how does each record describe its training scale and design?
• What do the evaluation papers in the library report about zero-shot use, scale and interpretability of these models?
• How far has a virtual-cell claim been tested: against held-out computational data, in a wet-lab assay, in animals, in patient data, or prospectively?
• Which library records show which kind of test, and where does the library say nothing?