Decomposing treatment effect estimation reveals asymmetric clinical knowledge in large language models: analysis code, frozen prompts and aggregate results

autoresearch/__init__.py

Package marker.

autoresearch/build_ground_truth.py

Builds the canonical ground-truth set, the train/validation split and component reference values.

autoresearch/enhanced_analysis.py

Additional analyses of harness outputs.

autoresearch/generate_cross_model_tables.py

Builds cross-model comparison tables from scorecards.

autoresearch/generate_tables.py

Builds comparison tables from a scorecard.

autoresearch/multi_model.py

Runs the frozen harness across model providers.

autoresearch/prompt_config.json

Frozen prompts, few-shot examples and generation settings for the three estimation strategies.

autoresearch/scorecard.py

Computes the optimization scorecard: ARR error, bias, limits of agreement, CCC and component metrics.

autoresearch/test_harness.py

Runs the three prompt strategies against a set of pairs.

scripts/01_scrape_thennt.py

Collects Tier 1 NNT and NNH values from TheNNT.com review pages.

scripts/02_extract_cochrane.py

Extracts Tier 2 NNT values from Cochrane systematic review abstracts through PubMed.

scripts/03_compute_ctgov_nnt.py

Computes Tier 3a NNTs from ClinicalTrials.gov trial results.

scripts/04_mimic_propensity.py

Computes Tier 3b NNTs by propensity-score matching in MIMIC-IV. Needs credentialed PhysioNet access.

scripts/05_harmonize.py

Harmonizes intervention and condition names across the four tiers.

scripts/06_llm_estimate.py

Queries the models for NNT estimates for each canonical intervention-condition pair.

scripts/07_analysis.py

Agreement, calibration and subgroup analyses.

scripts/08_hypothesis_generation.py

Exploratory hypothesis ranking and the MIMIC-IV comparison.

scripts/09_figures.py

Main and supplementary figures.

scripts/10_contamination_analysis.py

Training-data contamination analyses.

scripts/11_hypothesis_verification.py

openFDA and PubMed verification of the hypothesis pairs.

scripts/12_statistical_upgrades.py

CCC, ICC, Deming regression, mixed effects, kappa sensitivity, bootstrap intervals and multiple-testing correction.

scripts/13_restricted_analysis.py

Common-item restriction and tier-stratified agreement.

scripts/14_figures_v2.py

Supplementary figures added in revision.

scripts/15_arr_error_decomposition.py

First-order ARR-scale attribution of baseline event rate and relative risk errors.

results/analysis/arr_decomposition/arr_decomposition_results.json

Weighting factors and per-model baseline event rate and relative risk contributions to ARR error.

results/analysis/bland_altman/between_model_comparisons.json

Pairwise and cross-vendor tests of bias (paired t) and variance (Pitman-Morgan).

results/analysis/bland_altman/bland_altman_nonparametric.json

Non-parametric ranges of agreement per model and pooled.

results/analysis/bland_altman/bland_altman_results.json

Bias and limits of agreement per model, tier and model-tier, and pooled.

results/analysis/bootstrap_cis.json

Bootstrap 95% confidence intervals for MAE and rank correlation, per model and pooled.

results/analysis/calibration/calibration_results.json

Ordinary least squares calibration fits per model and pooled.

results/analysis/calibration/deming_results.json

Deming regression slopes and intercepts per model and pooled.

results/analysis/ccc/ccc_results.json

Lin's concordance correlation coefficient with bootstrap intervals.

results/analysis/contamination/contamination_summary.json

Summary of the three contamination analyses.

results/analysis/contamination/differential_contamination.json

Accuracy by evidence tier per model, with gradient tests.

results/analysis/contamination/rarity_stratification.json

PubMed hit-count tertiles and accuracy in each tertile.

results/analysis/contamination/temporal_holdout.json

Accuracy for evidence published before and after model training cutoffs.

results/analysis/contamination/verbatim_recall.json

Verbatim recall probe of TheNNT.com entries, with model responses.

results/analysis/extreme_discrepancies.json

The top 5% of absolute ARR differences by model, tier and direction.

results/analysis/icc/icc_results.json

ICC(2,1) and ICC(2,k) on the common-item set.

results/analysis/kappa/kappa_results.json

Weighted kappa per model and pooled.

results/analysis/kappa/kappa_sensitivity.json

Kappa under alternative category boundaries.

results/analysis/mixed_effects/mixed_model_results.json

Crossed random-effects variance components.

results/analysis/multiple_testing_correction.json

Holm-Bonferroni correction across the between-model tests.

results/analysis/restricted/common_item_results.json

Headline metrics on the common-item set against the full analysis.

results/analysis/restricted/tier_stratified_table.json

Agreement by model and evidence tier.

results/analysis/subgroups/confidence_calibration.json

Error by self-reported confidence, per model, with a Kruskal-Wallis test.

results/analysis/subgroups/confidence_deconfounded.json

Confidence signal after adjustment for tier and citation rarity.

results/analysis/subgroups/harm_subgroup_cis.json

Harm-direction subgroup metrics with confidence intervals.

results/analysis/subgroups/subgroup_results.json

Metrics by evidence tier, direction, therapeutic domain and confidence.

results/analysis/summary_tables.json

Model agreement summary table.

figures/fig1_reference_distribution.png

Reference NNT distribution by evidence tier.

figures/fig2_bland_altman_per_model.png

Bland-Altman agreement by model.

figures/fig_native_scale_error.png

Native-scale component error by model.

figures/fig_arr_error_attribution.png

First-order ARR-scale error attribution by model.

figures/supp_fig1_bland_altman_direction.png

Agreement by benefit or harm direction.

figures/supp_fig2_calibration_regression.png

Per-model calibration of estimated against reference ARR.

figures/supp_fig3_agreement_matrices.png

Five-category sensitivity classification by model.

figures/supp_fig4_confidence_calibration.png

Error by self-reported confidence.

figures/supp_fig7_contamination_temporal.png

Agreement by publication timing relative to training cutoffs.

figures/supp_fig8_contamination_rarity.png

Accuracy by citation rarity tertile.

figures/supp_fig_ber_rr_distributions.png

Reference baseline event rate and relative risk distributions.

figures/fig3_agreement_by_tier.png

Agreement metrics by model and evidence tier.

figures/fig4_hypothesis_predictions.png

High-agreement predictions before literature verification (exploratory).

requirements.txt

Python packages used by the code.