# Decomposing treatment effect estimation reveals asymmetric clinical knowledge in large language models: analysis code, frozen prompts and aggregate results

Version 1.0.1

Analysis code, frozen prompts and aggregate results for "Decomposing Treatment Effect Estimation Reveals Asymmetric Clinical Knowledge in Large Language Models" by Shuhan He, Amal Mohamed, Cory Rohlfsen, Brian W. Locke, Brian Sheridan, Natasha Akomeah, Alex Sorrick, Andrew S. Parsons, Ali Raja, Jarone Lee and David Liebovitz.

The study compared number-needed-to-treat (NNT) estimates from large language models with 539 reference values from four evidence tiers: TheNNT.com, Cochrane systematic reviews, ClinicalTrials.gov and MIMIC-IV. Models estimated treatment effects directly, as NNT or absolute risk reduction (ARR), and through a decomposed strategy that asks for the baseline event rate and then the relative risk. Prompts were developed on 374 training pairs and every result was confirmed on 165 held-out pairs. `METHODS.md` summarizes the design.

## Files

| Path | Contents |
| --- | --- |
| `autoresearch/prompt_config.json` | Frozen prompts and generation settings for the three estimation strategies |
| `autoresearch/*.py` | Estimation harness, scorecard, ground-truth builder and table generators |
| `scripts/01_*.py` to `scripts/15_*.py` | Analysis pipeline in execution order, from reference collection to figures |
| `results/analysis/` | 26 aggregate result files (JSON) behind the reported tables and figures |
| `figures/` | 13 figures reported in the paper and its supplement (PNG) |
| `METHODS.md` | Study design, reference tiers, strategies, models and statistics |
| `data_dictionary.csv` | What each code, prompt, result and figure file contains |
| `requirements.txt` | Python packages |
| `CITATION.cff`, `LICENSE` | Citation and licenses |

`MANIFEST.json` lists every file with its byte count and SHA-256.

## Patient data

No file contains patient-level data. MIMIC-IV was used under a PhysioNet credentialed data use agreement. Any language-model processing of MIMIC-IV data used a Microsoft Azure deployment in which data are not transmitted to the model provider, as PhysioNet permits for credentialed data. Figures derived from MIMIC-IV are not included. Rerunning `scripts/04_mimic_propensity.py` requires your own credentialed access to MIMIC-IV v2.2 through PhysioNet.

## Not included

Raw model-call records, scraped source text from TheNNT.com and Cochrane abstracts, and MIMIC-IV data or row-level derivatives are not redistributed. The reference sources are public and cited in `METHODS.md`.

## Running the code

The scripts ran on the authors' workstation, and paths near the top of several scripts still point there: the API key file, the MIMIC-IV database and a ClinicalTrials.gov export. Set them for your environment before running. Model calls need your own Anthropic, OpenAI and Google API keys. The scripts write results to `analysis/`; this release stores them under `results/analysis/`. You can read the results without rerunning any model.

## Versions

1.0.1 corrects the MIMIC-IV data-handling wording in `README.md` and `METHODS.md`. Code, prompts, results and figures are byte-identical to 1.0.0.

## Rights

Code in `scripts/` and `autoresearch/` (Python files) is released under the MIT License. Prompts, results, figures and documentation are released under CC BY 4.0. See `LICENSE`. Cite the dataset when you use the files; `CITATION.cff` gives the citation.