Episteme
Benchmarks · 18 Nov 2026

Does it work, and is the thesis right?

Benchmarks

Six benchmarks. Four check the components against external references: planted ground truth, OpenAlex, PubMed registry links and MeSH indexing. Two test the thesis against rigorous replication outcomes. The components mostly pass. The strong form of the thesis did not.

Thesis test: does pre-existing evidence predict ITP replication?

For 45 compounds with clear NIA Interventions Testing Program outcomes, we counted the aging literature published before each compound's ITP result, naively and as independent units. If the thesis held, the independent counts would separate later successes from failures (AUC above 0.5).

Naive (papers)Independent (units)dashed line = chance
ITP positiveITP nullhover for compound
Not supported. Neither raw nor independence-adjusted literature size predicts ITP success: AUC 0.40–0.48, and every interval includes chance. Heavily promoted compounds (resveratrol, metformin, fish oil, NR) failed, while acarbose succeeded with 3 prior papers. Attention is not evidence, and counting, even when it's corrected for duplicates, does not measure quality.
Exploratory signal (not pre-registered). Among compounds with at least 5 benefit-claiming papers, those that later failed had more duplicated claim literature (papers ÷ independent units): AUC –. Only 4 positives qualify, so this is a hypothesis to pre-register against the next ITP cohorts.

Findings layer: what the evidence said, read paper by paper

Claude subagents read 1,523 masked abstracts (compound hidden as COMPOUND-X) and recorded organism, context, endpoint and direction, each with a verbatim quote. The extraction holds up under audit. The pre-registered prediction still fails.

Not supported (pre-registered). Quality-weighted, independence-collapsed mammalian lifespan evidence does not predict ITP success: AUC –. The extraction is not the problem (14 of 16 audited correct, 0 wrong). The literature simply lacks the predictive evidence: most ITP successes had never been tested for lifespan in mammals before.

Synthetic literatures with planted labs

Adjusted Rand index between predicted evidence units and the true labs, averaged over 20 random literatures per scenario. 1.0 is perfect.

Same-trial detection

3,032 papers from 392 registered trials. Share of same-trial paper pairs that end up in the same unit.

The new named-trial rule ("the SPRINT trial") links pairs with 97% precision.

Design tier without MeSH

Held-out topic set (n = 437). Text-only tiering compared with the MeSH-based tier, recall per class, v0 → v1.

Author identity on real literatures

How often a name key ("wang j") is the same person across papers, according to OpenAlex. The table also shows what changes when v1 uses OpenAlex IDs.

Every number on this page is generated by benchmarks/*.py. Method details and caveats are in docs/BENCHMARKS.md.