TP53
98 variants0.472split-half PDS · n=59 scored · 95% CI 0.441–0.505
- Pairwise-resolvable pairs
- 0.0%
- Native-depth detection-eligible*
- 0%
- Top-10 split-half recovery
- 7.7% (chance 10.2%)
PertResolve / diagnostic framework
Distinguishing measurement and model limitations in perturbation prediction.
A benchmark cannot support biological distinctions that its own measurements do not reproducibly resolve.
PertResolve separates measurement questions from model questions, quantifies which perturbation distinctions reproduce in the experiment, and interprets prediction performance only at that supported resolution. It shows whether a prediction task needs better measurements or better models.
Framework overviewFigure 1
Core principle
Measure the distinction first. Interpret the prediction second.
01 / Framework
Measurement support and model performance should not be collapsed into one score.
PertResolve keeps measurement support and model performance separate. Detection asks whether a perturbation differs from reference; identification asks whether it differs from relevant competitors; split-half reproducibility asks whether the experiment recovers the same structure twice; model-ranking resolution asks which predictor gaps the benchmark can reliably order.
01 / Detection
Can the perturbation be distinguished from reference?
PertResolve measures whether a perturbation response separates reproducibly from the reference population under disjoint cell sampling.
Detection answers whether there is a measurable response. It does not imply that nearby perturbations can be identified.
02 / Study design
A fine-grained allele case and a broader perturbation panel answer complementary questions.
The allele-resolved arm tests whether experiments and models can distinguish variants of the same gene. The broader panel asks whether measurement-resolution constraints recur across genetic, chemical and cytokine perturbations, cellular contexts, donors and readout modalities.
470coding-variant conditions
321kcells in the allele-resolved arm
31perturbation configurations
14public data resources
| Allele dataset | Variants | Context | Assay |
|---|---|---|---|
| TP53 | 98 | A549 | Perturb-seq |
| KRAS | 92 | A549 | Perturb-seq |
| GATA1 | 254 | Human HSPCs | Base editing |
| JAK1 | 26 | HT-29 | scSNV-seq |
The scientifically relevant competitor is often a sibling perturbation, not an unrelated condition.
A model may recover a strong gene-level programme while failing to identify the correct allele. PertResolve therefore evaluates within-gene discrimination explicitly and then tests the same measurement logic across a broader perturbation landscape.
TP53KRASGATA1JAK1
03 / Fine-grained evidence
The experiment itself can fail to resolve the allele distinction being scored.
A disjoint split-half measurement provides an empirical reference for within-gene discrimination. TP53 and KRAS remain near chance, GATA1 shows partial structure, whereas JAK1 supports substantially stronger allele-level recovery. The four genes therefore expose different measurement regimes before any model is compared.
KRAS
92 variants0.481split-half PDS · n=51 scored · 95% CI 0.431–0.533
- Pairwise-resolvable pairs
- 0.0%
- Native-depth detection-eligible*
- 0%
- Top-10 split-half recovery
- 8.9% (chance 10.9%)
GATA1
254 variants0.578split-half PDS · n=172 scored · 95% CI 0.552–0.604
- Pairwise-resolvable pairs
- 3.8%
- Native-depth detection-eligible*
- 2.4%
- Top-10 split-half recovery
- 8.0% (chance 3.9%)
JAK1
26 variants0.807split-half PDS · n=26 scored · 95% CI 0.729–0.880
- Pairwise-resolvable pairs
- 78.5%
- Native-depth detection-eligible*
- 90%
- Top-10 split-half recovery
- 83.3% (chance 38.5%)
* Native-depth eligibility differs from the full variant count. The detection-eligible fractions use n=98 TP53, 92 KRAS, 254 GATA1 and 20 JAK1 variants evaluable at native depth; the split-half PDS and top-10 recovery use the scored counts printed above.
Near-chance split-half recovery changes what a model score can mean.
For TP53 and KRAS, two disjoint measurements of the same allele do not reliably recover within-gene identity. A near-chance model score in this regime cannot be interpreted simply as evidence that the model failed to learn a distinction that the experiment itself supports.
JAK1 separates measurement support from model capability.
The JAK1 split-half reference reaches 0.807, while the best feature-based predictor reaches 0.448, a gap of 0.36 in PDS. Here the experimental response contains reproducible allele structure, so the remaining prediction gap is not explained by the same measurement limitation.
Detecting an effect is easier than identifying the right sibling perturbation.
Pairwise resolvability ranges from 0% for TP53 and KRAS to 78.5% for JAK1. Fine-grained interpretation therefore depends on local competitor geometry rather than perturbation-versus-control separation alone.
Released frameworks
Released perturbation models show the same limit.
Released perturbation-prediction frameworks, evaluated on the same held-out variants with the same allele-level scoring, did not reliably identify alleles. Biolord, CellFlow and the allele-conditioned adaptation of scGen gave PDS values of 0.47–0.49 with confidence intervals overlapping 0.50.
PerturbNet reached PDS 0.530, above the uniform-matching expectation of 0.50 but not above its task-matched permutation null (null mean 0.515). scVIDR uses the held-out variant's measured response in its GATA1 scaling and is shown as a response-informed control.
| Released model | PDS | PDS 95% CI | Pearson-Δ |
|---|---|---|---|
| CellFlow | 0.493 | 0.467–0.520 | 0.156 |
| Biolord | 0.479 | 0.453–0.505 | 0.139 |
| scGenallele-conditioned adaptation | 0.474 | 0.449–0.500 | 0.201 |
| PerturbNettask-matched null 0.515 | 0.530 | 0.504–0.555 | 0.134 |
| scVIDRresponse-informed control | 0.486 | 0.461–0.511 | 0.183 |
470 → 4variant conditions → gene-level representations
Representation resolution must match evaluation resolution.
Gene-keyed interfaces such as scGPT, GEARS and STATE condition on perturbed-gene identity, so the 470 variant conditions collapse onto four conditioning keys and sibling alleles receive the same prediction. The Gene-mean reference, which predicts the mean response of a gene's training variants, gives PDS 0.478 (95% CI 0.450–0.504), indistinguishable from chance.
04 / Across-dataset generalization
Measurement resolution varies sharply across public perturbation resources and within the same resource.
The broader panel asks whether perturbation responses are reproducibly detectable across biological and technical regimes. Detection-side resolution spans the full 0–1 range, shifts across cell contexts and donors, and can differ between RNA and protein readouts.
lower detection resolutionhigher detection resolution
McFarland
GSE306429
CD34+A375A549
Tahoe-100M
Norman
Replogle
Adamson
Parse 10M
donor 1donor 12donor 5
KOLF2.1J
VCC
Papalexi
PerturbMulti
sci-Plex 3
Frangieh
X-Atlas
Fraction of perturbations meeting the detection criterion
05 / What shapes resolution
More cells help estimation, but they do not by themselves create biological separation.
Cross-dataset depth is an incomplete proxy for resolution. Controlled analyses instead point to the local separation between a perturbation and its closest competitors, together with sampling depth and benchmark size, as the quantities that determine what can be reproducibly distinguished and ranked.
Local competitor geometry is highly informative.
In the controlled resolution sweep, the median nearest-neighbour signal tracks the split-half discrimination reference more closely than the global signal summary. Fine-grained resolution is therefore governed by the closest confusable alternatives, not by average separation alone.
The deepest experiment is not necessarily the most resolvable.
Parse 10M is the deepest point in the cell-depth comparison, yet donor 1 reaches only 0.256 detection fraction. GSE306429 A549 reaches 1.00 with far fewer cells per perturbation. Depth reduces sampling uncertainty but cannot manufacture an absent or weak response distinction.
Resolution is not one scalar pass/fail property.
The controlled sweep shows monotonic improvement of the split-half discrimination reference with local signal, but model-order recovery is not itself monotonic. Whether a benchmark can rank predictors also depends on the score gap and number of evaluated conditions.
Pilot-based design
Averaged pilot scores rank perturbations by likely detectability.
In Replogle, Norman and Adamson, pilot scores from 25 cells per perturbation, averaged over ten pilot draws, ranked detectable above undetectable perturbations. A learned effect-size score and a training-free signal-to-variation rule both reach high AUROC. Identification among close competitors still requires assessing competitor geometry.
Pilot and evaluation cells are disjoint within each of ten draws. The detection label requires the median perturbation-to-reference distance, minus the median split-half distance, to exceed the 95% range of split-half distances at 100 cells per half.
| Dataset | Learned effect-size AUROC (25 cells) | Training-free signal-to-variation AUROC (25 cells) |
|---|---|---|
| Replogle | 0.960 | 0.934 |
| Norman | 0.916 | 0.851 |
| Adamson | 0.894 | 0.856 |
06 / Interpretation
The same prediction score has different meaning in different measurement regimes.
PertResolve does not replace prediction metrics with a composite resolution score. It supplies the measurement context needed to interpret them: whether the target distinction is experimentally supported, whether the model representation can express it, and whether the benchmark has enough resolution to order competing predictors.
Detectable is not identifiable.
A perturbation can differ clearly from control yet remain indistinguishable from its nearest biological competitors. Detection and identification therefore answer different measurement questions.
Measurement-limited is not model-limited.
TP53 and KRAS illustrate weak within-gene measurement recovery; JAK1 provides a contrasting regime where the experiment supports allele discrimination but current evaluated predictors still leave a large gap.
Benchmark ranking is its own problem.
Even when perturbation responses are measurable, a finite benchmark may not reliably order small model-score differences. Model-ranking resolution must therefore be assessed separately from perturbation detection or identification.
07 / Software
Apply the same diagnostic questions to your own representation.
The NumPy-first API accepts a cell-by-feature matrix or fixed representation and reports the measurement quantities separately rather than manufacturing a composite verdict.
From expression matrix to resolution report.
Use the bundled demo for a deterministic interface check, or provide your own perturbation labels and reference condition.
from pertresolve.resolution import resolution_report
report = resolution_report(
X,
labels,
control="non-targeting",
depth=50,
)
print(report.summary())Python ≥ 3.10 · NumPy / pandas core · MIT License
PPertResolve
2026
2026
Paper & resources
Distinguishing measurement and model limitations in perturbation prediction
The public repository includes the software, benchmark metadata, canonical result tables, analysis entry points and manuscript resources used by this project page.