Does it work, and is the thesis right?
Benchmarks
Six benchmarks. Four check the components against external references: planted ground truth, OpenAlex, PubMed registry links and MeSH indexing. Two test the thesis against rigorous replication outcomes. The components mostly pass. The strong form of the thesis did not.
Thesis test: does pre-existing evidence predict ITP replication?
For 45 compounds with clear NIA Interventions Testing Program outcomes, we counted the aging literature published before each compound's ITP result, naively and as independent units. If the thesis held, the independent counts would separate later successes from failures (AUC above 0.5).
Findings layer: what the evidence said, read paper by paper
Claude subagents read 1,523 masked abstracts (compound hidden as COMPOUND-X) and recorded organism, context, endpoint and direction, each with a verbatim quote. The extraction holds up under audit. The pre-registered prediction still fails.
Synthetic literatures with planted labs
Adjusted Rand index between predicted evidence units and the true labs, averaged over 20 random literatures per scenario. 1.0 is perfect.
Same-trial detection
3,032 papers from 392 registered trials. Share of same-trial paper pairs that end up in the same unit.
The new named-trial rule ("the SPRINT trial") links pairs with 97% precision.
Design tier without MeSH
Held-out topic set (n = 437). Text-only tiering compared with the MeSH-based tier, recall per class, v0 → v1.
Author identity on real literatures
How often a name key ("wang j") is the same person across papers, according to OpenAlex. The table also shows what changes when v1 uses OpenAlex IDs.
Every number on this page is generated by benchmarks/*.py. Method details and caveats are in docs/BENCHMARKS.md.