# Methods

Summary of the study design behind this release. The paper, "Decomposing Treatment Effect Estimation Reveals Asymmetric Clinical Knowledge in Large Language Models", gives the full methods.

## Reference data

Reference NNTs came from four tiers:

- Tier 1: 457 values from TheNNT.com, collected in February 2026.
- Tier 2: 1,384 values from PubMed-indexed Cochrane systematic review abstracts. For 264 pairs, LLM-assisted parsing extracted control-group event rates and pooled relative risks, which serve as component reference values.
- Tier 3a: 503 values computed as 1/ARR from arm event rates of registered trials with posted results on ClinicalTrials.gov.
- Tier 3b: 13 values from propensity-matched critical-care outcomes in MIMIC-IV v2.2.

The 2,357 values cover 576 intervention-condition pairs. The 539 pairs with a canonical reference NNT were split 70/30 with seed 42, stratified by tier and ARR magnitude, into 374 training and 165 held-out validation pairs.

## Estimation strategies

- Direct NNT: the model estimates NNT for an intervention-condition pair.
- Direct ARR: the model estimates ARR directly.
- Decomposed: the model estimates the baseline event rate (BER), then the relative risk (RR) with its own BER as context; ARR = BER × (1 − RR).

All strategies return structured JSON. The exact prompts and settings are in `autoresearch/prompt_config.json`.

## Models

The direct-estimate panel was Claude Opus 4.6, Claude Sonnet 4.6, GPT-4o, o3-mini, Gemini 2.5 Flash and Gemini 2.5 Pro. The decomposition analysis used Claude Sonnet 4.6, GPT-4o, GPT-5 and Gemini 2.5 Pro. Standard models ran at temperature 0; reasoning models used their native reasoning modes.

## Prompt optimization

Ten experiments on the training set with Claude Sonnet 4.6 varied templates, few-shot examples and calibration instructions. The best configuration was frozen and applied to every model. The validation set was never used during optimization.

## Statistical analysis

Continuous analyses use signed ARR (ARR = ±1/NNT, positive for benefit). Reported measures include mean absolute error, signed bias, Bland-Altman analysis with non-parametric 95% ranges of agreement, Lin's concordance correlation coefficient (10,000 bias-corrected and accelerated bootstrap replicates), Spearman correlation and bootstrap intervals for MAE (5,000 replicates), and tier accuracy for strong (NNT 10 or less), moderate (over 10 to 50) and weak (over 50) treatments. Further analyses cover Deming regression, crossed random-effects variance decomposition, ICC(2,1), Holm-Bonferroni correction, contamination checks (temporal holdout, citation rarity, verbatim recall) and a first-order attribution of ARR error to BER and RR errors.

## Data handling

MIMIC-IV was accessed under a PhysioNet credentialed data use agreement. Any language-model processing of MIMIC-IV data used a Microsoft Azure deployment in which data are not transmitted to the model provider, as PhysioNet permits for credentialed data.