# Tahoe-100M: can held-out glioma lines' drug responses be predicted from the other lines? — GlioState own analysis (draft)

Run 2026-10-19/20 (Asia/Taipei). Exploratory re-analysis of public data; not peer-reviewed; no deep-learning model evaluated.

## Data (not redistributed)
- Hugging Face `tahoebio/Tahoe-100M`, revision `349e2a55f5538a3e73b32a06c6bdd8e9e87011e8` (lastModified 2026-10-14), licence CC0-1.0.
- `metadata/obs_metadata.parquet` (sha256 e51b24f3…29d4d, 100,648,790 rows) → `cell_counts.json` (00_cell_counts.py). Brain lines: A-172 2,352,722; H4 1,075,515; SW 1088 163,938; CCF-STTG1 0 (in cell_line_metadata only). Together 3.57% of all rows (3.67% of pass_filter = full cells). 50 lines carry data.
- Official pseudobulk DE table `metadata/pseudobulk_differential_expression/train-*-of-01026.parquet` (all 1,026 shards, ~89 GB streamed; per-shard sha256 in `shard_sha256.json`). Columns baseMean / log2FoldChange / lfcSE / padj per gene × (plate, line, drug, concentration), treated vs plate-matched DMSO. The README does not document the DE method; we treat it as DESeq2-style output as labelled. 65,218 contrasts after merging the 1,025 contrasts split across shards.
- Raw cell-level shards `data/train-XXXXX-of-03388.parquet`: the 1,694 even-numbered shards (50% of shards; shards are per plate with cells shuffled across wells, so this is an approximately random half of cells), ~178 GB streamed, not stored (04_raw_pseudobulk.py).
- Other metadata (sha256): cell_line 67641f5b…, drug 7a04c7a6…, gene d6104d1c…, sample 33167f0c….

## Steps (venv: numpy, scipy, pandas, pyarrow)
1. `01_fetch_de.py OUT 6` — stream DE shards → compact npz (log2FC/lfcSE float16, baseMean float32); resumable. ~26 min.
2. `02_assemble.py` — merge split contrasts; baseline proxy per line = mean log1p(CP10k of baseMean) over its contrasts (a proxy for control expression, NOT the DMSO profile); "expressed" genes = CP10k ≥ 0.2 in ≥ 45/50 lines (5,136). ~4 min. `02b_panel.py` — scoring panel = 2,000 expressed genes with largest log2FC signal variance (mean lfc² − lfcSE²) over NON-glioma contrasts (`panel_hvg2000.txt`); raw panel adds 1,000 most variable baseline genes + markers (`panel_raw.txt`, 2,427 genes).
3. `04_raw_pseudobulk.py OUT panel_raw.txt 2 0 5` — per cell: drop CLS entry, CP10k, log1p; sums per (well, line, half), half = crc32(barcode) mod 2. ~32 min.
4. `05_raw_assemble.py` — control = all DMSO_TF cells of that line on that plate (only 2 DMSO wells per plate); delta = condition mean − control for all cells, half 0 (truth) and half 1 (technical duplicate; shares the control); conditions kept if each half ≥ 20 cells; DMSO null = each DMSO well vs the other DMSO well(s) of the same plate; per line "effective" = mean delta² > 95th percentile of that null. 25,004 of 52,371 line×condition pairs effective (A-172 526/1,123; H4 404/1,114; SW 1088 30/528).
5. `03_analysis.py` three runs: `results_de.json` (DE source, effective = mean (lfc/lfcSE)² ≥ 2, which is nearly everything and too lenient), `results_de_effmask.json` (DE source, effective mask from step 4), `results_raw.json` (cell-level source). ~7 min each. `06_report.py` → `report_*.txt`.

## Task and predictors (03)
Truth = response vector (log2FC for DE; log1p delta, half 0, for raw) of a held-out line for one drug × dose (1,135 DE / 1,137 raw conditions); 2,000-gene panel. Conditions with < 50 treated cells (DE) are dropped; NCI-H596 and NCI-H2122 are skipped entirely (median 13 and 21 treated cells), leaving 48 evaluable lines.
- LOGO: train on the 47 non-glioma lines, predict each glioma line. LOO: every line held out, trained on the other 49 (ranking).
- Predictors: training mean (per condition); nearest 1 / 5 lines by baseline correlation (2,000 most variable baseline genes); ridge on 10 baseline PCs (λ by inner leave-one-line-out over a grid, chose 100 in DE runs); shuffled-feature ridge null (10 permutations); random training line (expected value); same-organ mean; for glioma, mean of the other two glioma lines; in-context variants that see the held-out line on a random half of drugs (kNN-5 by response correlation; mean × fitted amplitude), scored on the other half; duplicate = cross-plate replicate (DE; 6,939 pairs) or half-1 (raw).
- Metrics: MSE, Pearson over genes. 95% CIs: bootstrap over drugs (2,000 resamples; all doses/lines of a drug together). Noise floor = mean lfcSE² (DE) or (h0−h1)²/2 (raw).

## Main results (LOGO, 3 glioma lines pooled; all conditions)
| Source | Predictor | MSE | Δ MSE vs mean [95% CI] | Pearson |
|---|---|---|---|---|
| DE (n=3,149) | training mean | 0.3748 | – | 0.574 |
| | ridge PCA10 | 0.3719 | −0.0029 [−0.0034, −0.0024] | 0.581 |
| | NN-5 baseline | 0.3819 | +0.0071 [+0.0062, +0.0080] | 0.555 |
| | other-two-glioma mean | – | +0.1815 [+0.1751, +0.1882] | Δr −0.208 |
| | cross-plate replicate (n=406) | 0.6394 | +0.342 | 0.252 |
| Raw (n=2,765) | training mean | 0.0090 | – | 0.654 |
| | ridge PCA10 | 0.0086 | −0.0004 [−0.0004, −0.0003] | 0.672 |
| | NN-5 baseline | 0.0087 | −0.0003 [−0.0004, −0.0003] | 0.666 |
| | other-two-glioma mean | – | +0.0027 [+0.0025, +0.0028] | Δr −0.096 |
| | half-cell duplicate | 0.0094 | +0.0004 [0.0000, +0.0007] | 0.712 |
Effective conditions (raw, n=960): mean r 0.721, ridge 0.743 (ΔMSE −0.0007 [−0.0009, −0.0006]), duplicate 0.820 (ΔMSE −0.0030).
LOO rank by training-mean Pearson (1 = hardest of 48): DE SW 1088 1, H4 13, A-172 33; raw SW 1088 3, H4 14, A-172 24. The hardest lines are those with fewest cells.

## Caveats
- Tahoe's Mosaic design pools all lines in the same well, so the training mean shares each well's technical effect with the truth; the cross-plate replicate does not (hence its low score). The training mean is therefore optimistic relative to transfer to a separately run glioma model.
- Raw duplicate shares the control with the truth; raw source uses 50% of shards. Only 2 DMSO wells per plate, so the DMSO null is coarse.
- DE baseline is a baseMean proxy; marker values come from it. Ridge/NN are simple re-implementations; no deep model; no GSC, organoid, patient-derived or in vivo data.