Linear baselines and systematic variation in perturbation prediction: what 'a strong baseline' means, and where the papers disagree

Research question

• When papers report that a mean or linear baseline matches deep models, are they comparing the same object, with the same metric, on the same kind of perturbations?
• Which mechanisms could make a perturbation-agnostic predictor look good (shared response, noise, target gene, metric), and which papers actually test which?
• What would the claim 'deep models do not beat simple baselines' need in order to hold, and what would the counter-claim need?

Current understanding

Our thesis: 'a simple baseline matches deep models' is not one finding but at least four, and the papers that report it define the baseline, the metric and the perturbation panel differently. Reading the full texts, we separate four mechanisms that can each make a perturbation-agnostic predictor look good: a shared response across perturbations (VC0052), noise and bias–variance in the control (VC0145), triviality of the target gene (VC0049, VC0144) and metrics that dilute weak signals (VC0146). No paper tests all four on one dataset, and the one prominent counter-position (VC0146, authors employed by the company Shift Bioscience) argues that under calibrated metrics deep models do beat uninformative baselines. Our own pseudobulk analysis (a re-implementation; no model evaluated), repeated over 20 random perturbation-level splits, is consistent with parts of both sides. On the public 2025 H1 data a linear model built from training data was worse than the training mean on MSE and control-referenced Pearson delta in every split and only marginally better than a shuffled-embedding null; setting only the target gene lifted Pearson delta from 0.16 to 0.34; and a technical duplicate beat the mean on MSE in the single split for 89% of perturbations, unlike the roughly 5% reported on K562 genome-wide data. On Replogle K562 and RPE1 essential-gene pseudobulks the same linear model matched the mean on MSE, scored higher on retrieval metrics than both the mean and the null in all 20 splits, and with 900 training perturbations also beat the mean on MSE, so our earlier H1-only reading that the linear model "did not beat the training mean on any metric" does not generalise. Certainty: high that the papers state what we attribute to them; moderate for our computations as computed; low for generalising them to other datasets or to any model. Scope: method evaluation only; not related to glioma or any clinical question.

Scope and how to read this page. This page is a commentary on how perturbation-prediction papers define baselines and metrics; it is not a leaderboard and does not rank models. For the Virtual Cell Challenge's own metrics see the companion page (/validation/virtual-cell-challenge/); we cross-refer to its numbers rather than repeat them. Certainty labels say how well the cited sources, or our reproducible computation, support a claim, not how important it is. 'Our own analysis' means a GlioState re-implementation on public pseudobulk data, not peer reviewed. Relevance to glioma or other tumours: none (not applicable); all evidence concerns CRISPR perturbation screens in cell lines, and we draw no tumour, tissue or clinical inference. VC0048 VC0052

A1 — 'The baseline' is a different object in each paper. Claim: results labelled 'mean baseline' are not interchangeable. Evidence from the full texts: VC0048 'mean' = mean of the training perturbation profiles (each perturbation counts once) and, separately, a linear model Y ≈ G·W·Pᵀ + b with PCA-derived embeddings (ridge λ = 0.1); VC0052 'perturbed mean' = mean of all perturbed training cells (each cell counts once); VC0049 'Train Mean' = mean of the training pseudobulk profiles; VC0144 'CRISPR-informed mean' = perturbed-cell training mean with the target gene overwritten (0 for CRISPRi, 2× the training mean for CRISPRa); VC0145 mean and median of the training targets, inside a regression framework; VC0146 a control mean (8,192 control cells), the perturbation-weighted mean (it states that this weights every perturbation equally rather than every cell), a technical duplicate, an interpolated duplicate and the VC0048 linear model. Our own analysis on H1 (below) shows the weighting choice is small there (cosine of the two mean deltas 0.92; MSE 3.410 vs 3.417 ×10⁻³) but the target-gene rule and the reference choice are not (A4). Certainty: high that this is what the papers define; the size of the differences is dataset-dependent and we measured it on one dataset only. VC0048 VC0052 VC0049 VC0144 VC0145 VC0146

A2 — Four mechanisms can each make a perturbation-agnostic predictor look good, and the papers test different ones. (i) Shared response: VC0052 defines systematic variation as the cosine between each perturbation's shift and the average shift, reports means of 0.76 (Adamson), 0.50 (Norman) and 0.32 (Replogle K562), and that dataset-level systematic variation correlates strongly with Pearson delta of GEARS (0.95) and fine-tuned scGPT (0.91); VC0145 states the same idea as 'stereotypical responses' and tests it by correlating treatment-over-control changes from independent experiments with independent controls (positive and far from zero in some datasets). (ii) Noise and bias–variance: VC0145 argues the control mean is unbiased but noisy, whereas the all-training mean is biased by perturbation effects but lower-variance, and shows in simulated noise-only data that the training mean can give correlations above 0.6 with observed changes. (iii) Target-gene triviality: VC0049 shows scGPT's advantage over the training mean in the top-20-DE-gene metric shrinks when the target gene is removed from that list, and VC0144 builds this knowledge into a baseline; VC0145 sets the target gene by construction and scores only other genes. (iv) Metric dilution: VC0146 argues that for datasets with few DE genes per perturbation (3.25 on average in K562 genome-wide vs 102.09 in Norman) the technical duplicate loses to the mean on MSE, so MSE cannot reveal signal. Our reasoning: (i) and (ii) are not mutually exclusive and are hard to separate without replicated controls; (iv) is about the metric, the others about the data. No paper varies one mechanism while holding the others fixed on one dataset. Certainty: high for attribution; the claim that the mechanisms are untested jointly is our reading and could be wrong where supplementary material (not read) does it. VC0052 VC0145 VC0049 VC0144 VC0146

A3 — Our own analysis: on the public 2025 H1 data, a linear model built from training data did not beat the training mean, and its embedding carried at most weak perturbation-specific information; on K562 and RPE1 essential-gene data the picture is different. (Single-split numbers first; the 20-split update follows.) Setup (details under Methods and results): VC0048's linear model re-implemented on pseudobulk, trained on 150 perturbations, scored on 150 held-out ones (validation 50 + test 100) against half-cell truth. Results (pooled over the 150, target gene excluded): MSE 3.41 ×10⁻³ for the training mean vs 3.86 for the linear model with K = 10 and 4.54 for K = 50; L2 on the 1,000 control-top genes 2.26 vs 2.54; Pearson delta (control reference) 0.148 vs 0.103; PDS-L1 0.510 vs 0.510 and PDS-L2 0.562 vs 0.557. The linear model was better than the mean on MSE for only 17% of perturbations (paired bootstrap of the mean MSE improvement: −4.5 ×10⁻⁴, 95% interval −5.8 to −3.2 ×10⁻⁴). A K = 1 model matched the mean, larger K was worse, and giving each held-out perturbation another perturbation's embedding (20 shuffles) gave MSE 3.90, Pearson delta (training-mean reference) −0.010 against 0.015, with paired intervals for the real-minus-shuffled difference including zero. Our reading: with 150 training perturbations and weak shared structure (mean cosine 0.17 on this panel, companion page), the embedding-to-response map is not learnable by this model; this does not say the model fails on K562/RPE1 data, where VC0048 reports it does well. Certainty: moderate that these numbers follow from our code; low that they say anything about other data; possible that a different PCA convention than the authors' changes the linear-model result (their code was not run). Update 2026-10-17, 20 random perturbation-level splits (H1 has only 300 perturbations, re-partitioned 150/150 each time, so splits overlap; the cell split stays fixed): H1 MSE 3.74±0.44 (training mean), 4.04±0.39 (linear K=10), 4.61±0.31 (K=50), 4.10±0.39 (shuffled null) ×10⁻³; Pearson delta 0.161±0.012, 0.111±0.015, 0.089±0.012, 0.100±0.012; PDS-L2 0.553, 0.552, 0.556, 0.552. The linear model was worse than the mean on MSE and Pearson delta in 20/20 splits, as in the single split. Differences from the earlier text: K=10 is slightly better than its shuffled null on Pearson delta (+0.011±0.010; 18/20 splits) and MSE (−0.07×10⁻³), so "no detectable information" should read "very weak information". On Replogle K562 and RPE1 essential-gene pseudobulks (150 training / 150 held-out, 20 splits) linear K=10 matches the mean on MSE (3.39 vs 3.36 and 7.98 vs 7.86 ×10⁻³) and is about 0.01 lower on Pearson delta, but beats the mean and the shuffled null on PDS-L1/L2 in 20/20 splits (by about 0.02); with 900 training perturbations it also has lower MSE (19–20/20 splits). So "a linear model does not beat the mean" holds on H1 and on error/correlation metrics at 150 training perturbations, not in general (Tables 5–6). VC0048

A4 — The reference and the target gene can move Pearson delta more than the choice between a mean and a linear model. Claim: Pearson-delta values are only comparable when the reference and the target-gene handling are stated. Evidence (our own analysis, same 150 held-out perturbations): the training-mean predictor scores Pearson delta 0.148 against the control reference; predicting the control instead is undefined against that reference, but against the training-mean reference the control mean scores 0.243 and the cell-weighted mean 0.208, i.e. a 'no change' predictor scores higher than a perturbed-mean-like predictor under a perturbed-reference metric, which we do not interpret beyond noting that the two references reward different things. The technical duplicate's Pearson delta is 0.65 against the control reference and 0.74 against the training-mean reference (target excluded), so the achievable ceiling is itself reference-dependent. Overwriting only the target gene (median training knockdown ratio 0.095 × control) moves the mean predictor's Pearson delta from 0.145 to 0.345 and the linear model's from 0.107 to 0.257 (target included), while MSE and PDS barely change. This agrees in direction with VC0049's top-20-gene observation and with the premise of VC0144's CRISPR-informed mean; VC0052 reports its perturbed-centroid reference gives scores with low correlation to systematic variation (0.21 vs the standard metric's high correlation). Certainty: moderate (our computation; the target-gene effect is simple and reproducible; the 'control mean beats cell-weighted mean under a perturbed reference' pattern is observed once and we do not explain it). Update 2026-10-17: over 20 splits the overwrite lifts the mean predictor from 0.156 to 0.344 (H1), but only from 0.231 to 0.253 (K562) and 0.475 to 0.481 (RPE1) (Table 7), so the size of the target-gene effect is dataset-dependent. VC0049 VC0144 VC0052

A5 — Calibration is dataset-dependent: a technical duplicate is a good positive control on some datasets and a poor one on others. Claim: whether MSE or Pearson delta 'fail' depends on perturbation strength and cell numbers, as VC0146 itself says. Evidence: VC0146 reports the technical duplicate underperformed the mean on MSE in about 95% of perturbations in Replogle K562 genome-wide data (3.25 DE genes per perturbation on average) but outperformed it in Norman (102.09). Our own analysis on H1 (median 1,045 cells per training perturbation; half-cell truth and duplicate): the technical duplicate beat the training mean on MSE for 89% of 150 perturbations; the dynamic range fraction (VC0146 definition, technical duplicate as positive control, training mean as negative) averaged 0.48 for MSE, 0.59 for the L2 on control-top-1000 genes, 0.61 for Pearson delta (control reference), 0.48 for Pearson delta (training-mean reference, control mean as negative control), 0.84 for PDS-L1 and 0.87 for PDS-L2 (median close to 1 for both PDS). So on H1 the unweighted metrics are partly calibrated and the retrieval metrics more so; this does not reproduce VC0146's K562 pattern and we could not test its weighted MSE or interpolated duplicate, which need a DE step. Our reading: the stated disagreement between 'MSE is uninformative' (VC0146) and 'simple baselines win on MSE' (VC0048) is partly a statement about which dataset and perturbation panel was used. Certainty: moderate for the H1 numbers; low for the K562 contrast, which compares our half-cell pseudobulk with VC0146's own setup. Update 2026-10-17: across 20 H1 splits the technical duplicate has MSE 1.13±0.11 against 3.74±0.44 ×10⁻³ for the training mean; the K562/RPE1 files used for the repeat have no half-cell truth, so no duplicate could be scored there and VC0146's K562 contrast remains untested by us. VC0146 VC0048

A6 — Retrieval-type scores separate 'informative' from 'mean-like' predictions more sharply than error or correlation, with a caveat about amplitude. Claim: the centroid accuracy of VC0052 and a PDS-L2 on deltas are the same kind of quantity, and both sit near chance for every perturbation-agnostic predictor we tried. Evidence: VC0052 defines centroid accuracy as the fraction of other perturbations whose centroid is farther (Euclidean) from the predicted profile than the true centroid; our PDS-L2 is 1 − rank/N of the true perturbation among all evaluated ones by L2 distance of the predicted to the real deltas. Our observation, not a statement of either paper: with a shared control these differ only in gene exclusion and in which perturbations are pooled. On H1 the mean-type predictors, the linear model at every K and the shuffled null score 0.55–0.59 (chance 0.5), the technical duplicate 0.999; VC0146 reports its strongest correspondence between neighbourhood recovery and retrieval-type metrics (ρ > 0.78). The caveat from the companion page applies: L1-type retrieval scores also depend on prediction amplitude, so a high retrieval score is not a pure 'right direction' score. Certainty: moderate for the equivalence (our reading of the definitions), moderate for the H1 numbers. Update 2026-10-17: on K562 and RPE1 the K=10 linear model scores PDS-L2 0.530 and 0.530 against 0.511 and 0.507 for the training mean, so retrieval scores are the metrics that react to the embedding there; on H1 they do not (0.552 vs 0.553). VC0052 VC0146

B1 — Where the conclusions conflict, and where they do not. VC0048, VC0049, VC0052, VC0144 and VC0145 report that simple baselines match or beat the deep models they tested (scGPT, scFoundation, GEARS and others; VC0145 tests mainly interpretable regression methods and GEARS); VC0146 reports the opposite for calibrated metrics. They are not strictly contradictory: VC0146's models partly differ (PRESAGE, scLambda, CellFlow and embedding probes were not in the older benchmarks), it states that earlier models such as scGPT and GEARS do not beat baselines under MSE and Pearson delta in K562, consistent with the older papers, and that on every metric at least one deep model outperforms all baselines (including the VC0048 linear model) on its Replogle K562 data (Supplementary Table 2, not read by us). The real disagreement is about which metrics are admissible evidence. Two cautions. First, VC0146's authors are employees of Shift Bioscience, and one co-author is at Xaira Therapeutics (its competing-interest statement); we did not check the competing-interest statements of the other papers. This is a disclosure, not a judgment of the work. Second, VC0053 (a 2026 bioRxiv preprint, not peer reviewed) re-evaluates VC0146's benchmark and reports that scGPT's weighted-MSE advantage arose from how gene subsets were handled; we did not reproduce this and treat it as an unreplicated claim. Certainty: high for what each paper reports; low-to-moderate for which side is right, which we do not settle. VC0048 VC0049 VC0052 VC0144 VC0145 VC0146 VC0053 · preprint

B2 — Perturbation selection can decide how strong the baseline looks, and no paper varies it cleanly. Evidence: VC0052 attributes systematic variation to selection biases or confounders, and calls for 'heterogeneous' gene panels (its Adamson panel has mean cosine 0.76); VC0049 reports that Adamson and Replogle RPE1 show high similarity between perturbation profiles, making models hard to distinguish, and that its benchmark covers only perturbation-exclusive (PEX), not cell-exclusive (CEX), prediction; VC0145 notes that in most of its 11 datasets the perturbed genes were chosen for survival or regulatory importance, with Replogle genome-wide as the one exception, and analyses TF-only and large-effect subsets; VC0146 notes weak perturbations in K562 genome-wide (median 1 DE gene) and HepG2 (0), stronger ones in Wessels (8); VC0048 states four datasets, all cancer cell lines, and no filtering for perturbations that did not reduce their own target. The 2025 Challenge H1 panel was curated by the organisers (companion page; its mean cosine is 0.17). Our reasoning: panel curation (essential genes, knockdown filters) changes both the shared response and the signal-to-noise per perturbation, so a baseline's strength is a property of the panel and not only of the method. Certainty: high that these selections are as stated; moderate for the inference. VC0052 VC0049 VC0145 VC0146 VC0048

B3 — Three claims that are often run together should be kept apart. (1) 'A simple baseline beats model X on metric M on dataset D' is what the papers show. (2) 'Model X has not learned perturbation-specific biology' needs, in addition, a metric shown to detect such learning and an upper bound for what is achievable; VC0146 and VC0053 argue for both, and VC0145 calls it a possibility that held-out perturbation prediction may not be feasible from a sample of perturbations. (3) 'The task is infeasible' is stated by none of them in this strong form; VC0048 says its baselines cannot represent real biological complexity and that prediction remains an open challenge, and VC0048's own linear model with embeddings pretrained on perturbation data did consistently beat all other models, whereas pretraining on atlas data gave only a small benefit over random embeddings. Our reasoning: most of the public disagreement is between (1) and (2). Certainty: high that the papers say what is stated here; the separation is our editorial framing. VC0048 VC0145 VC0146 VC0053 · preprint

C — What each claim needs in order to be true (conditions, not findings). (C1) 'Deep models do not beat simple baselines': the baselines must be defined and, ideally, released (mean weighting, reference, target-gene rule); the metrics must be shown to be sensitive on that dataset (a positive control that beats the baseline); the panel must not be so homogeneous that no method could separate; scores should come with perturbation-level uncertainty. (C2) 'Deep models do beat baselines on calibrated metrics': the calibration must hold on the dataset used (A5 shows dataset dependence); the weighting or DE step must not itself leak information or hide gene-subset effects (the point of dispute with VC0053); the models compared should be the same under every metric family. (C3) 'Systematic variation explains the strong mean baseline': needs the cosine measure to be computed on the same panel as the benchmark, a reference that removes it without making weak perturbations noisy (VC0052 itself warns that reference-sensitive metrics can be noisy for weak perturbations), and a test against the noise explanation. (C4) 'A linear model built from training data is a fair baseline': needs the embedding to be informative on that panel (our shuffled-embedding null is a cheap check), and sensitivity to K and the PCA convention reported. Certainty: these are our editorial conditions, labelled as reasoning. VC0048 VC0052 VC0053 · preprint VC0146

What would change our view. (a) The same nine-or-so models and the baselines of VC0146, scored under all four conditions on one dataset where the technical duplicate clearly beats the mean (as on H1), with the target gene fixed and both references reported: if deep models then beat both the mean and the linear model on error-type metrics, our reading of B1 as 'mostly metric choice' would need revising towards 'dataset choice'. (b) A controlled subsampling of one genome-wide dataset into strong, weak, essential-only and random panels, holding model and metric fixed: it would show how much baseline strength is panel-driven (B2). (c) Running the authors' own code of VC0048 on H1: if it beats the mean there, our linear-model null in A3 reflects our PCA convention rather than the data. (d) An independent replication of VC0053's claim about gene-subset handling in VC0146: if it holds, part of the evidence for 'models beat baselines' would weaken; if it fails, the counter-position would strengthen. (e) Reading the unread supplements (VC0146 Supplementary Notes 1–2 and Tables 1–2; VC0145 Additional Files 2–3; VC0052 supplementary figures): they may already contain some of the tests we call missing in A2. (f) Replicated-control data showing that the stereotyped-response correlation of VC0145 is near zero on a panel where a mean baseline still wins: it would point to noise rather than shared biology. Thresholds are our judgment. Nothing here would extend the scope: results concern CRISPR perturbation screens in cell lines, with no tumour or clinical inference. VC0146 VC0053 · preprint VC0145 VC0052 VC0048

Certainty summary (draft grading, GS-GRADE track C: benchmark claims). A1, A2, B1–B3: attribution read from the full text of VC0048, VC0049, VC0052, VC0144, VC0145 and VC0146 (high that the papers state what is reported; none of this is a judgment of correctness); VC0051 and VC0053 are preprints, entered at abstract level in the comparison table, and VC0053's claim about VC0146 is unreplicated. A3–A6: our own computation on pseudobulk; single-split numbers on H1 plus a 20-split repeat on H1 (300 perturbations only, overlapping splits), K562 and RPE1 (Replogle 2022 essential-gene bulk files, different normalisation and gene set), cell split not repeated, no model evaluated; official cell-eval run once on a separate single split (Table 8), not on the repeat (moderate for the numbers, low for generalisation, and the H1-only conclusions were revised as described in A3). C and the 'what would change our view' list: editorial conditions. One of the cited items (VC0146) comes from authors employed by a company (Shift Bioscience; competing-interest statement in the paper); two (VC0051, VC0053) are preprints. VC0048 VC0049 VC0052 VC0144 VC0145 VC0146 VC0051 · preprint VC0053 · preprint

Last updated: 2026-10-17 (Pacific Time). Multi-split repeat added; draft commentary pending review; no deployment. To be revisited when the unread supplements are read and when VC0053 or an independent group reports on VC0146's benchmark.

Evidence map

Primary studies & datasets

• VC0048 Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines. — Ahlmann-Eltze C, Huber W, Anders S., Nature methods, 2025
• VC0052 Systema: a framework for evaluating genetic perturbation response prediction beyond systematic variation. — Viñas Torné R, Wiatrak M, Piran Z, Fan S, Jiang L, Teichmann SA, Nitzan M, Brbić M., Nature biotechnology, 2025
• VC0144 Simple controls exceed best deep learning algorithms and reveal foundation model effectiveness for predicting genetic perturbations. — Wong DR, Hill AS, Moccia R., Bioinformatics, 2025
• VC0145 A comparison of computational methods for expression forecasting. — Kernfeld E, Yang Y, Weinstock JS, Battle A, Cahan P., Genome biology, 2025
• VC0146 Deep learning perturbation models can outperform baselines on calibrated metrics. — Miller HE, Mejia GM, Leblanc FJA, Swain B, Wang B, de Lima Camillo LP., Nature biotechnology, 2026

Perturbation & validation evidence

• VC0049 Benchmarking foundation cell models for post-perturbation RNA-seq prediction. — Csendes G, Sanz G, Szalay KZ, Szalai B., BMC genomics, 2025
• VC0051 · preprint PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction — Wenteler A, Occhetta M, Branson N, Curean V, Huebner M, Dee W, Connell W, Chung SP, Hawkins-Hooker A, Ektefaie Y, Valdez Córdova CM, Gallagher-Syed A., bioRxiv, 2024
• VC0053 · preprint Signal, Bounds, and Baselines: Principles for Evaluating Virtual Cell Perturbation Models — Vollenweider M, Bühlmann P., bioRxiv, 2026

8 library items cited on this page; 2 are preprints (not peer reviewed).

Entities on this page

• CRISPR interference Methods & concepts
• GEARS Models
• Simple linear baseline Methods & concepts
• Perturb-seq Methods & concepts
• Perturbation response prediction Methods & concepts
• scFoundation Models
• scGPT Models
• Systematic variation Methods & concepts
• Virtual Cell Challenge Methods & concepts

Methods and results (our own analysis)

This is an exploratory re-analysis of public data run by GlioState, not a peer-reviewed result. Outputs: linear_baselines/results.json · linear_baselines/README.md · linear_baselines/multi_split_summary.json · linear_baselines/cell_eval_results.json (the analysis script itself is not published on this site)

Data, code and caveats

Data: the same public 2025 Virtual Cell Challenge H1 hESC CRISPRi Perturb-seq pseudobulks as the companion page (train 150 perturbations; validation 50; test 100; 18,080 genes; control 'non-targeting', 38,176 control cells in the training file); not redistributed. Code: analysis/linear_baselines/03_linear_baselines.py (numpy and scipy; about 2 minutes CPU) on the .npz files written by analysis/vcc_baseline/01_pseudobulk.py. Truth for every predictor is the half-0 pseudobulk of each held-out perturbation, so the technical duplicate (half 1) is on the same footing. Caveats: our re-implementation of VC0048's linear model (PCA convention is our reading), pseudobulk only, no DE-based metrics, no deep model, one cell line, bootstrap over perturbations only (Tables 2–4: one random cell split). Update 2026-10-17: Tables 5–7 repeat the comparison over 20 random perturbation-level train/held-out splits (analysis/linear_baselines/04_multi_split.py, 06_summarize.py) on H1 and on the public Replogle 2022 K562 and RPE1 essential-gene pseudobulks (figshare 20029387, raw_bulk files; per-guide-set mean UMI counts, CP10k, log1p; 79.8 and 95.4 MB). H1 offers only 300 perturbations (150 train + 50 validation + 100 test), so each split re-partitions the same 300 into 150 training and 150 held-out perturbations: splits overlap heavily and the SD describes sensitivity to which perturbations are held out, not sampling from a wider population. The cell-level split (H1: seed-0 halves) was not repeated. Replogle files have no half-cell truth, so truth equals the pseudobulk used elsewhere and there is no technical duplicate there. This is our own exploratory analysis, not a peer-reviewed result.

Table 1. How each paper defines baselines, metrics and perturbation selection, and what it does not test (our reading of the full texts; not a computation)

Reading Table 1. Baselines: only VC0146 and VC0048 include a linear model; only VC0146 includes a positive control (technical and interpolated duplicates); VC0144 is the only one that builds the target gene into the baseline, while VC0145 handles it by construction and VC0049 only by removing it from one metric. Metrics: Pearson delta appears in VC0048, VC0052, VC0049, VC0144 and VC0146 but with different references (control in most, perturbed centroid in VC0052, both in VC0146); error metrics appear in VC0048 (L2 on 1,000 genes), VC0145 (MAE) and VC0146 (MSE and weighted MSE); only VC0052 and VC0146 use retrieval-type scores. Perturbation selection: the GEARS-processed Adamson, Norman and Replogle sets recur in VC0048, VC0049, VC0052 and VC0144, so their conclusions are not independent evidence about new panels. The last column lists only what we could ground in the texts; blanks do not mean a paper is flawed. VC0048 VC0049 VC0052 VC0144 VC0145 VC0146

Table 2. Predictors on 150 held-out H1 perturbations (pooled validation 50 + test 100; half-cell truth; target gene excluded)

Table 3. Target-gene handling and paired comparisons (same 150 perturbations)

Table 4. Dynamic range fraction on H1 (VC0146 definition; technical duplicate positive, train mean negative)

Reading the tables. (1) In Table 2 the linear model never exceeds the training mean on MSE, L2 or control-referenced Pearson delta, and its larger-K versions are worse; the shuffled-embedding null is not distinguishable from K = 10 in Table 3. (2) The retrieval scores of every non-duplicate predictor sit at 0.51–0.57, close to the 0.5 expected for a perturbation-agnostic prediction, while the technical duplicate reaches 0.96–1.00. (3) Table 3 shows the target-gene effect on Pearson delta (0.145 to 0.345). (4) Table 4's dynamic range fractions are well above zero on H1, unlike the K562 genome-wide pattern VC0146 describes for MSE. These are our own pseudobulk numbers; see the caveats above and under Uncertainties. Full outputs: analysis/linear_baselines/results.json. Update 2026-10-17 (Tables 5–7): over 20 perturbation-level splits the H1 pattern of Tables 2–3 holds for MSE and control-referenced Pearson delta (linear K=10 worse than the training mean in 20/20 splits on both), but not for everything we first wrote: the K=10 model is slightly better than its shuffled null on Pearson delta (18/20 splits, +0.011±0.010) and on MSE (by 0.07×10⁻³), so the embedding is not wholly uninformative on H1, only weakly so; and on K562 and RPE1 the linear model scores higher than both the training mean and the null on retrieval scores in 20/20 splits and, with 900 training perturbations, also on MSE (19–20/20). The target-gene overwrite effect (Table 7) is large on H1 and small on K562/RPE1. VC0146

Table 5. 20 random perturbation-level splits: mean ± SD of each metric (target gene excluded; truth for H1 = half-0 pseudobulk, for K562/RPE1 = the pseudobulk)

Table 6. Paired differences over the 20 splits (mean ± SD of per-split difference; in brackets the number of splits in which the stated direction holds)

Table 7. Target-gene handling, Pearson delta (control reference), mean ± SD over 20 splits

Reading Tables 5–7. (1) H1: on every split the linear model has higher MSE than the training mean (K=10: +0.30×10⁻³; K=50: +0.87×10⁻³) and lower control-referenced Pearson delta; retrieval scores for all predictors stay at 0.51–0.56 against 0.5 chance, with K=50 slightly above the mean on PDS-L1 (+0.006±0.006). The technical duplicate reaches MSE 1.13×10⁻³ and PDS-L2 0.99 on average over the splits, so the metrics can separate informative from uninformative predictions on H1. (2) K562 and RPE1 (Replogle essential-gene panel, 150 training perturbations): linear K=10 matches the training mean on MSE (K562 +0.03×10⁻³, 7/20 splits lower; RPE1 +0.12×10⁻³, 6/20 lower) and is slightly lower on Pearson delta (−0.010), yet its PDS-L1/L2 exceed the mean and the shuffled null in 20/20 splits (by about 0.02), so here the embedding carries perturbation-specific information that the error and correlation metrics hardly reward. With 900 training perturbations K=10 also lowers MSE below the mean (K562 −0.1×10⁻³, 20/20; RPE1 −0.3×10⁻³, 19/20) and raises Pearson delta on K562 (+0.023, 20/20). The training mean itself is strong on this panel (control-referenced Pearson delta 0.23 on K562, 0.48 on RPE1) and clearly beats the control mean on MSE (RPE1: 7.9 vs 10.9×10⁻³), consistent with a large shared response among essential-gene knockdowns. (3) Table 7: overwriting the target gene lifts the mean predictor from 0.156 to 0.344 on H1 but only from 0.231 to 0.253 on K562 and 0.475 to 0.481 on RPE1; the effect depends on how much of the signal is the target gene itself, and the Replogle sets use 8.5–8.7 thousand expressed genes against 18,080 on H1. (4) What changed relative to the single-split reading: the H1-only statements (linear no better than mean on MSE, L2 and Pearson delta; K=1–5 approximately equal) are reproduced, but "no detectable perturbation-specific information in the embedding" is softened to "very weak on H1", and the generalisation that a linear model does not beat the mean is not supported on K562/RPE1 retrieval metrics or with more training perturbations. Caveats: the two datasets differ in technology, normalisation, gene set and perturbation panel as well as in cell line, so differences between H1 and K562/RPE1 cannot be attributed to the cell line; the five shuffles per split give a noisier null than the 20 shuffles of Tables 2–3. Full outputs: analysis/linear_baselines/multi_split_*.json and multi_split_summary.json. VC0048 VC0146

Table 8. Official cell-eval 0.8.2 (profile full), one split: 50 held-out H1 perturbations (training-file cells), predictors fit on 100 others

Reading Table 8. The package installed and ran without modification (pip install cell-eval; predicted profiles replicated per cell as in its baseline construction; linear predictions clipped at 0 because it rejects negative log1p values; ≤100 real cells per held-out perturbation, 1,000 control cells). Our pseudobulk MSE and Pearson delta computed from the same files match cell-eval's to four digits (training mean: Pearson Δ 0.2477, MSE 4.31×10⁻³), which supports our re-implementation of those two metrics. The ordering agrees with Tables 2 and 5: training mean best on MSE and Pearson delta, linear worse, K=10 and its shuffled null equal; on PDS the shuffled null is even above K=10 on this split, and the linear models have higher DE significant-gene recall (0.58–0.63 vs 0.56), which we did not analyse further. One split of 50 perturbations with small cell numbers is not comparable to the 150-perturbation tables and has no uncertainty interval; we did not run the official leaderboard configuration or the ceiling option. A first cell-eval run was discarded because of a labelling bug in our preparation script (found and fixed before any number was used). VC0146

What remains uncertain

Unread or only partly read sources. Not read: VC0146 supplementary notes, tables and extended data (we relied on the main text and Methods); VC0145 Additional Files 2–3 (derivations and supplementary figures; we cite the simulation and the stereotyped-response test as described in the main text); VC0052 supplementary figures and tables; VC0048 extended data; the PertEval-scFM peer-reviewed version (VC0051 was entered from the bioRxiv record at abstract level, and the table row says so); VC0053 beyond its abstract-level claim and the VC0146 re-evaluation point; the original GEARS, scGPT and scFoundation papers (we do not assess their own claims); PerturBench (cited by VC0144), Bendidi et al., Liu et al., Nadig et al. and Martens et al. (cited by the papers above but not fetched, so not in the knowledge base). VC0146 VC0145 VC0052 VC0048 VC0051 · preprint VC0053 · preprint

Our own analysis is a pseudobulk re-implementation on one dataset (2025 H1 hESC CRISPRi, 150 training perturbations, 150 held-out, half-cell truth, one fixed random cell split; the 20-split repeat varies only which perturbations are held out, and because H1 has just 300 perturbations the 20 splits overlap and their SD understates variability across new data). The PCA convention for the linear model's G matrix is our reading of VC0048's Methods and the authors' code was not run. No differential-expression metric (weighted MSE, DEG-restricted Pearson, interpolated duplicate) and no deep model were evaluated, so the H1 results cannot speak to VC0146's main claim. Perturbed-centroid referencing as in VC0052 was approximated with the perturbation-weighted training mean. Exact verification steps: (1) obtain the public H1 files and run analysis/vcc_baseline/01_pseudobulk.py then analysis/linear_baselines/03_linear_baselines.py; (2) run VC0048's published code (github.com/const-ae/linear_perturbation_prediction-Paper) on the same split and compare K = 10 against our linear-model rows; (3) obtain Replogle K562 essential data in the GEARS format, run VC0052's and VC0146's published code, and compute the technical-duplicate-versus-mean MSE comparison per perturbation; (4) repeat A3–A5 with the official cell-eval pipeline for the DE-based metrics. Additional limitations of the 2026-10-17 repeat: (a) the K562 and RPE1 files are Replogle 2022 essential-gene bulk pseudobulks (about 1,800 perturbations whose target gene is among 8.5–8.7 thousand measured genes, at least 25 cells; per-guide-set mean UMIs re-normalised to CP10k and log1p by us), so differences from H1 reflect technology, normalisation, gene set and panel as well as cell line; (b) these files have no half-cell truth, so truth and training come from the same pseudobulk and there is no technical duplicate or noise ceiling; (c) the shuffled null uses 5 shuffles per split; (d) we did not run the authors' code, so the linear-model result may still depend on our PCA convention; (e) the official cell-eval package (0.8.2) was run only once, on a different single split (100 training / 50 held-out perturbations from the H1 training file, ≤100 real cells each, linear predictions clipped at 0), not on the 20-split repeat (Table 8); (f) we computed no DE-based metrics. VC0048 VC0052 VC0146

The four-mechanism split in A2 is our synthesis; the papers do not frame their results this way, and some mechanisms may overlap (a shared response can be partly a control artefact, as VC0052 and VC0145 both suggest). The claim that no paper tests all four on one dataset rests on the main texts we read. We did not quantify how much of H1's strong-mean behaviour is shared biology versus noise; the replicated-control test of VC0145 was not run because the H1 files have a single, large control pool. VC0052 VC0145

Everything on this page concerns evaluation of perturbation-prediction methods on CRISPR screens in cell lines (H1 hESC here; K562, RPE1, HepG2 in the cited papers). It is not related to glioma or to any tumour, tissue or clinical question (not applicable), and nothing here supports a claim about either.

Related questions

• How are virtual cell models validated?
• Do deep learning models beat linear baselines for perturbation prediction?
• What do the cell-eval metrics used by the Challenge actually reward, when checked against code and public data?
• Where do the two main baseline critiques (VC0048, VC0052) agree, conflict, or rest on unstated assumptions?