Virtual Cell Challenge metrics and baselines: what the scores reward, and where the evidence conflicts

Research question

• What do the cell-eval metrics used by the Challenge actually reward, when checked against code and public data?
• Where do the two main baseline critiques (VC0048, VC0052) agree, conflict, or rest on unstated assumptions?
• What would each common claim about the benchmark need in order to be true?

Current understanding

Our thesis: the Challenge's scores reward narrower things than their names suggest, and the two best-known baseline critiques do not explain each other. From the cell-eval code and our own pseudobulk computation on the public 2025 H1 data (a re-implementation, not the official pipeline): a perturbation-agnostic prediction scores about 0.51 on L1 PDS, PDS rises with prediction amplitude while MAE falls, Pearson delta can be doubled by setting one gene, and the baseline-normalised score clips at zero. VC0048 and VC0052 differ in baseline definition, metric and data, and VC0052's mechanism does not account for VC0048's error-based result. No 2026 results exist; our computed numbers are 2025 in-context, not zero-shot (certainty: moderate for the code reading and computations, low for transfer to 2026).

Scope. This page does not retell the Challenge. One line of context: the 2026 round asks for zero-shot CRISPRi response prediction in six unseen cell lines, scored with Arc's cell-eval; final test set 2026-10-22, submissions due 2026-11-05, no results as of 2026-10-16 (Arc, VC0107). We ask three narrower questions we could check against primary sources: what the metrics reward (A), where VC0048 and VC0052 conflict (B), and what each claim needs to be true (C). Certainty labels refer to how well the cited sources, or our reproducible computation, support a claim, not to its importance. Items marked 'our own analysis' are re-implementations on pseudobulk data with the caveats listed under Methods and results. VC0107

A1 — L1 PDS rewards amplitude as well as direction, and a perturbation-agnostic prediction sits near 0.5. Claim: the discrimination score cannot be read as a pure 'right direction' score. Evidence (cell-eval code, commit 6928967, 2026-07-27): for each perturbation, the predicted delta from control is compared by L1 distance with the real deltas of all perturbations in the evaluation set (target gene excluded), and the score is 1 − rank/N. Our own analysis (public 2025 H1 data; validation 50 perts, test 100 perts): predicting the control mean, or the mean of 150 training perturbation means, scores 0.507–0.514 PDS-L1; a split-half noise ceiling scores 0.96. Multiplying a noisy-copy delta by 0.1 / 0.5 / 1 / 2 / 4 moves PDS-L1 from 0.53 to 0.74 to 0.965 to 0.986 to 0.992, while Pearson delta (0.74) and cosine-PDS (0.9996) do not change, and MAE worsens from 0.0178 to 0.0394 to 0.093 beyond 1×. Our reasoning: a model that shrinks uncertain predictions toward the mean (which lowers error) is penalised by PDS, and one that inflates them is rewarded until saturation, so PDS and MAE pull in opposite directions. This agrees qualitatively with Arc's report that a participating team found L1 PDS scale-sensitive (VC0105; we did not read that team's commentary). Certainty: moderate that the numbers are right for this re-implementation (reproducible, one seed, noisy copies not model outputs); low for any statement about 2026 entries. VC0105

A2 — The baseline-normalised score clips at zero, so a metric no entry can beat the baseline on carries no ranking information. Claim: this is a plausible mechanism for Arc's statement that MAE stopped influencing the 2025 leaderboard. Evidence (cell-eval `_score.py`): for error metrics the score is 1 − user/base, for bounded metrics (user − base)/(1 − base); NaN becomes 0, values are clipped at 0, and the final score is the mean. Our own analysis: on MAE the control mean (0.0255 val / 0.0240 test) beats the mean-of-perturbation-means baseline (0.0266 / 0.0250), and even the split-half noise ceiling (0.0176 / 0.0187) yields a normalised MAE score of only 0.34 / 0.25, whereas PDS-L1 at the ceiling gives 0.93 / 0.92. So MAE can contribute at most a fraction of its nominal weight on this data, and swapping the baseline from the perturbation mean to the control mean, or to a cell-weighted mean (0.0258 / 0.0242), shifts the MAE score by roughly 0.03–0.04. Whether the Challenge leaderboard used exactly this function and which baseline it used is not stated in the sources we read; the link to VC0105 is our inference. Certainty: moderate for the code reading, low for the link to the leaderboard. VC0105

A3 — Pearson delta over ~18,000 genes can be doubled by one gene. Claim: unless the target gene's handling is stated, Pearson-delta numbers are not comparable across entries. Evidence (cell-eval code): PDS excludes the target gene by default (`exclude_target_gene=True`); `pearson_delta` does not. Our own analysis: a predictor that adds the training mean delta to the control raises Pearson delta from 0.166 to 0.395 (val) and 0.170 to 0.378 (test) when the target gene alone is set to the median training knockdown level (0.095× control); MAE moves from 0.0263 to 0.0262 and PDS does not move. Pearson delta was one of four extra metrics in the 2025 Generalist Prize ranking (VC0105); which metrics the 2026 panel uses was not read by us. Certainty: moderate (own computation, simple and reproducible; all 300 target genes are in the 18,080-gene panel). VC0105

A4 — On the 2025 H1 panel the shared-direction component is small, so baseline strength found elsewhere should not be assumed here. Claim: by a VC0052-style measure, systematic variation is low in this panel. Our own analysis: the mean cosine between each perturbation's delta and the mean delta of the 150 training perturbations is 0.17 (val) / 0.17 (test); leave-one-out within the evaluation set 0.13 / 0.16; only 5–6% of the squared delta norm lies along the training mean-delta direction. Mean-type predictors score Pearson delta 0.15–0.17 and PDS-L1 0.51, against noise ceilings of 0.73–0.74 and 0.96. VC0052 reports mean cosines of 0.76 (Adamson), 0.50 (Norman) and 0.32 (Replogle K562) on its own processing (5,000 highly variable genes plus perturbed genes). Those are not directly comparable to ours (we use all 18,080 genes, so noise lowers our cosine). This fits Arc's stated design aim of a panel selected for diverse responses (VC0002: 300 perturbations chosen from about 2,500 for strength and diversity of response; VC0143 describes the 2026 curation). Our reasoning: Arc's 2025 remark that models were not consistently beating naive baselines (VC0105) then has to be read as depending on which baseline and metric (we found MAE to be the weak one), not as evidence that a strong mean baseline exists on this panel. Certainty: low-to-moderate. VC0002 VC0143 VC0052 VC0105

B1 — What the sources actually did (attribution checked against the full texts). VC0048 (Ahlmann-Eltze, Huber, Anders; Nature Methods 2025): five foundation models and two other deep-learning models against a 'no change' and an 'additive' baseline for double perturbations (Norman, CRISPRa, K562), and against a 'mean' baseline and a linear model for unseen single perturbations (Replogle K562 and RPE1, both CRISPRi; Adamson K562); main error measure L2 distance on the 1,000 most highly expressed genes, Pearson delta secondary. 'Mean' = mean of the training perturbation profiles. It does not analyse systematic variation. Its linear model with a perturbation embedding pretrained on the other Replogle cell line did consistently beat all other models, so the finding is that the deep models did not beat simple ones, not that nothing does. The authors state limits: four datasets, all cancer cell lines, no knockdown quality filtering, and that their baselines cannot represent real biological complexity. VC0052 (Viñas Torné et al.; Nature Biotechnology 2025): three methods (CPA, GEARS, scGPT) against a 'perturbed mean' (mean of all perturbed training cells) on ten datasets; Pearson delta on all genes and on top-20 DE genes; systematic variation measured as cosine between a perturbation shift and the average shift. The claim that systematic variation inflates reference-based scores is VC0052's, not VC0048's. Correction to the earlier draft of this page, which blended the two. VC0105 is Arc's own account; the PDS-scale analysis is attributed there to a participating team and the 'PDS about twice DES' weighting to the second-place team's own description, neither of which we verified independently. Certainty: high that this is what the papers state (full text read for VC0048 and VC0052); not a statement about correctness. VC0048 VC0052 VC0105

B2 — Where the conclusions conflict or rest on unstated assumptions. (1) Metric dependence: VC0052 says its inflation mechanism affects reference-based metrics such as Pearson delta, and that RMSE did not show the same dependence on systematic variation; yet VC0048's headline result is under an error measure (L2) and simple baselines still won. So VC0052's mechanism cannot be the whole explanation for VC0048. Our reasoning: error-type metrics leave little room above a mean-type predictor because much of the error is shared level and noise; on H1 the noise-ceiling MAE is 0.0176 against 0.0255 for predicting control (val), i.e. about 69% of it. (2) The baseline is not one object: VC0048's mean weights perturbations equally, VC0052's perturbed mean weights cells, cell-eval's baseline builder uses the mean of per-perturbation means. On the H1 training set cells per perturbation range from 33 to 4,760; the two means' deltas have cosine 0.92 and give MAE 0.0266 vs 0.0258 (val), 0.0250 vs 0.0242 (test). (3) Pre-selection: VC0052 keeps 1,973 strong-effect Replogle perturbations; VC0048 restricts to perturbations all models could predict; the 2026 data are curated to at least 80% median knockdown (VC0143). All three assume conclusions carry from effect-filtered panels to unfiltered ones. (4) Shared response as artefact or biology: VC0052 itself lists biological causes (stress, cell-cycle arrest) among the sources of systematic variation, yet treats removing it (perturbed-centroid reference) as isolating 'perturbation-specific' effects, and states that the reference-based metrics are invariant to effect strength and unreliable for weak perturbations. (5) Neither paper tests prediction in an unseen cell line without any perturbation data; the 2026 task does (VC0107). VC0048 reports a hint that transfer between two lines helps more for genes similar across lines, from a single pair. Certainty: high for what each paper states; low-to-moderate for our reconciliation (our reasoning). VC0048 VC0052 VC0143 VC0107

C — What each claim would need to be true. (C1) 'A top-ranked 2026 entry captures perturbation biology.' Needs: PDS above the ~0.5 perturbation-agnostic level that survives amplitude control (norm-matched or cosine variants agree); DE-based metrics above a constant-gene-set baseline (we did not compute these); per-metric results rather than one aggregate; and robustness when the target gene is excluded. (C2) 'Models do not beat naive baselines' (Arc 2025, VC0105). Needs: the baseline stated (perturbation-weighted, cell-weighted or control), the same normalisation as the entries, and a metric not already at its noise ceiling; none is stated in what we read, so the claim is currently not checkable. (C3) 'Deep models do not beat linear baselines' (VC0048). Needs: the result to hold when the panel is not effect-filtered, under tuning budgets comparable to the baselines (the authors capped fine-tuning at three days and used default parameters), and in the setting that matters here: an unseen cell line. (C4) 'Baselines win because of systematic variation' (VC0052). Needs: baseline success to fall when the shared component is removed under a metric that is robust to it, a reconciliation with error-based results, and the shared component to be non-biological; our single H1 panel, with low shared direction and weak mean predictors, is consistent with it but cannot test a relationship that VC0052 estimates across ten datasets. (C5) 'A good 2026 score means a model transfers across cellular contexts.' Needs: per-line results on the three held-out lines, a stated baseline built from other-line data, and a check that the held-out lines' responses are not already covered by public training data. Nothing we computed bears on C5. Certainty: our reasoning throughout (conditions, not findings). VC0105 VC0048 VC0052 VC0107

What would change our view. (a) A full `cell-eval run --profile full` on the baselines below (not done by us; steps under Uncertainties): if a constant predictor scores well on DE-overlap metrics, the 'agnostic ≈ 0.5' picture of PDS would not extend to DES. (b) Arc publishing the 2026 baseline definition and per-metric, per-line results: this would settle C1, C2 and C5 directly; a top entry that beats the baseline on every metric and every held-out line would weaken our concern, wins on one metric only would strengthen it. (c) Reading the Team Outlier commentary on PDS scaling and the BioMap and Altos methods: if their scaling is calibrated differently from our noisy-copy demonstration, A1 needs revision. (d) An independent re-analysis of the 2025 leaderboard. (e) Evidence that the shared response in H1 or the 2026 lines is biological and predictable: it would turn the shared component from a nuisance into something worth rewarding. Thresholds are our judgment. Nothing here would change the scope limits: CRISPRi knockdown in cell lines only, no drug, primary-cell, tissue or clinical inference. VC0105 VC0107

Certainty summary (draft grading, GS-GRADE track C: benchmark claims). A1–A4: code reading from a fixed public commit of the organiser's tool (moderate; the leaderboard may have run a different version) plus our own reproducible computation (moderate for the numbers, low for 2026). B1–B2: attribution and methods read from the full text of VC0048 and VC0052 (high that they state what is reported); VC0105 and VC0002 are organiser-authored, VC0002 read only as a search excerpt of the Cell perspective's figure legend. The 2026 Cell paper (VC0058) was verified by record and abstract only. C: conditions, not findings. Claims marked 'our reasoning' are editorial judgments. None of the cited items is a preprint. VC0048 VC0052 VC0105 VC0002 VC0058

Last updated: 2026-10-16 (Pacific Time). Draft commentary pending review; to be revisited when the 2026 final results and winners' methods are public.

Evidence map

Primary studies & datasets

• VC0002 Virtual Cell Challenge: Toward a Turing test for the virtual cell. — Roohani YH et al., Cell, 2025
• VC0058 Virtual Cell Challenge 2026: Benchmarking zero-shot generalization across cellular contexts. — Arc Virtual Cell Initiative Team, Goodarzi H., Cell, 2026
• VC0107 Virtual Cell Challenge 2026 (zero-shot, 6 unseen cell lines) — Arc Institute, virtualcellchallenge.org, 2026
• VC0143 Generating high-quality Perturb-seq data for the 2026 Virtual Cell Challenge — Arc Institute, Arc news, 2026

Perturbation & validation evidence

• VC0105 Virtual Cell Challenge 2025 Wrap-Up: Winners and Reflections — Arc Institute, Arc news, 2025
• VC0048 Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines. — Ahlmann-Eltze C, Huber W, Anders S., Nature methods, 2025
• VC0052 Systema: a framework for evaluating genetic perturbation response prediction beyond systematic variation. — Viñas Torné R, Wiatrak M, Piran Z, Fan S, Jiang L, Teichmann SA, Nitzan M, Brbić M., Nature biotechnology, 2025

7 library items cited on this page; 0 are preprints (not peer reviewed).

Entities on this page

• CPA Models
• CRISPR interference Methods & concepts
• Single-cell foundation model Methods & concepts
• GEARS Models
• Simple linear baseline Methods & concepts
• Perturb-seq Methods & concepts
• scGPT Models
• Systematic variation Methods & concepts
• Virtual Cell Challenge Methods & concepts
• Zero-shot evaluation Methods & concepts

Methods and results (our own analysis)

This is an exploratory re-analysis of public data run by GlioState, not a peer-reviewed result. Outputs: vcc_baseline/results.json · vcc_baseline/README.md (the analysis script itself is not published on this site)

Data, code and caveats

Data: public Arc Virtual Cell Atlas, 2025 H1 hESC CRISPRi Perturb-seq (train 150 perturbations / 221,273 cells; validation 50 / 98,927; test 100 / 170,846; 18,080 genes; 'non-targeting' control), downloaded 2026-10-16 from the public bucket; not redistributed. Code: our scripts in analysis/vcc_baseline/ (stream the CSR matrix, scanpy-default normalisation and log1p, per-target mean = pseudobulk, metrics re-implemented from cell-eval commit 6928967). Caveats: pseudobulk re-implementation, not the official pipeline; DE-based metrics not computed; no model evaluated; 2025 in-context task; own-file median normalisation, no `cell-eval prep` gene filtering; split-half ceiling uses half the cells and no Spearman–Brown correction; single seed, no confidence intervals. This is our own exploratory analysis, not a peer-reviewed result. VC0143

Table 1. Trivial predictors under cell-eval metrics (pseudobulk)

Table 2. Amplitude sensitivity (noisy-copy delta × factor, validation)

Table 3. Shared direction and score headroom

What remains uncertain

Our computation is a pseudobulk re-implementation of cell-eval metrics on the 2025 in-context task. It is not the official `cell-eval run`, no differential-expression metric (the DES counterpart, overlap/precision, AUC, direction match) was computed, and it says nothing about 2026 zero-shot results. Proposed verification, exact steps: (1) `uv pip install cell-eval`; (2) download adata_Training.h5ad, adata_Validation.h5ad and pert_counts_Validation.csv from the public 2025 bucket; (3) `cell-eval baseline --adata adata_Training.h5ad --counts pert_counts_Validation.csv --pert-col target_gene --control-pert non-targeting --output-path baseline_pred.h5ad --num-threads 16`; (4) `cell-eval run -ap baseline_pred.h5ad -ar adata_Validation.h5ad --pert-col target_gene --control-pert non-targeting --profile full --ceiling --outdir out_base`; (5) repeat with a control-mean prediction; (6) compare `agg_results.csv` with `cell-eval score`. Memory for differential expression on about 99,000 cells is the practical constraint (we did not attempt it).

The 2026 six-metric panel, its weights and the baseline Arc will use were not read (the Challenge website did not render for our fetch tool). Statements about 'the Challenge metrics' in A1–A3 rest on the cell-eval repository (its `vcc` profile lists MAE, L1 discrimination score and overlap-at-N; that overlap-at-N is the DES counterpart is our inference) and on the 2025 wrap-up. VC0105 VC0107

The PDS-scale commentary by the participating team (arXiv), the BioMap, XLearning, Outlier and Altos methods, and the full text of the 2026 Cell paper (VC0058) were not read. No independent evaluation of the dataset or scoring exists in the library. VC0105 VC0058

Single random seed for the split-half ceiling and scale demonstration, 50 and 100 perturbations, no confidence intervals. Systematic-variation cosines use all 18,080 genes and are biased downward by noise.

Our judgments about metric behaviour and baselines are reasoning from code, our computation and two papers on other datasets; they are not findings about 2026 entries and could be wrong. VC0048 VC0052

Related questions

• How are virtual cell models validated?
• Do deep learning models beat linear baselines for perturbation prediction?
• When papers report that a mean or linear baseline matches deep models, are they comparing the same object, with the same metric, on the same kind of perturbations?
• Which mechanisms could make a perturbation-agnostic predictor look good (shared response, noise, target gene, metric), and which papers actually test which?