The Evidence Accessibility Problem

Randomized controlled trials are widely regarded as the gold standard for causal inference in the social sciences. Over the past two decades, their growing adoption in development economics has produced thousands of rigorous evaluations spanning education, health, agriculture, social protection, governance, labor markets, and financial inclusion across more than 150 countries. Repositories such as J-PAL, 3ie, and IPA, along with major economics journals, have amassed a vast and growing body of experimental evidence on what works in global development.

Yet this wealth of knowledge remains remarkably difficult to access and use. The quantitative findings of each study are not stored in a database; they are buried inside lengthy PDF documents, scattered across tables with inconsistent formatting, and reported using a wide variety of statistical conventions. A single RCT may contain dozens of treatment arms, hundreds of outcome measurements, and multiple econometric specifications, with key information distributed across narrative text, dense results tables, and supplementary figures. Reconstructing the full experimental design of even one study, identifying which interventions were tested, which arms were compared, what outcomes were measured, and what effect sizes were observed, is a labor-intensive process that domain experts currently perform manually.

The consequences are significant. Policymakers seeking to understand the average impact of a class of interventions must rely on systematic reviews that are expensive, slow to produce, and quickly outdated as new trials are published. Researchers attempting meta-analyses face months of extraction work before analysis can begin. And billions of dollars in development aid continue to be allocated based on incomplete evidence, not because rigorous evidence does not exist, but because it is locked inside documents that no one has the time to read.

Figure 1. The extraction challenge. A single RCT document contains interventions described in narrative text, treatment arms that bundle multiple interventions, and outcomes with effect sizes scattered across results tables. Our pipeline identifies these entities across modalities and reconstructs the full hierarchical structure linking them.

Our Approach: A Multi-Stage Extraction Pipeline

Each RCT encodes a deep dependency chain: interventions map to treatment arms, arms form counterfactual contrasts, and effect sizes link each contrast to specific outcomes and populations. A single linking error cascades through the entire structure. Our pipeline addresses this by decomposing extraction into sequential stages, each building on verified outputs of the previous one.

Phase 1: Document Processing and Table Filtering

The pipeline begins by ingesting full-text RCT papers and converting them into a structured representation. We use OCR-based parsing to extract text and tables, with specialized handling for rotated tables and complex layouts. Figures are converted to text descriptions. The result is a unified document representation that preserves all the information a human reader would have access to.

Not all tables in a paper contain treatment effects. Most contain baseline statistics, balance checks, descriptive summaries, or robustness analyses. To identify the tables that matter, we apply a multi-stage filtering pipeline that progressively narrows the set of candidate tables to only those reporting main treatment effect estimates on the full experimental sample, using the best available econometric specification.

In parallel, we score all text sections in each paper for topical relevance across multiple dimensions: measures and outcomes, sampling and study design, intervention details, econometric methods, and implementation quality. This relevance scoring ensures that each downstream extraction stage receives the most informative context from the paper, not just the tables, but the surrounding narrative that describes how the study was designed and conducted.

Phase 2: Structured Entity Extraction

With the relevant tables and text identified, the pipeline reconstructs the hierarchical experimental design through a series of coordinated extraction stages. Each stage uses large language models prompted with domain-specific instructions developed in close collaboration with development economics experts.

To make this concrete, consider what the pipeline produces from a single study:

Phase 3: Effect Size Standardization

A central challenge in building a cross-study database is that different RCTs report their results in different units: raw mean differences, regression coefficients, odds ratios, risk differences, or pre-standardized effect sizes. To enable meaningful comparison and aggregation, every extracted effect size is converted to a common metric: Hedges' g, a bias-corrected standardized mean difference.

The standardization process follows a decision tree that handles continuous outcomes (converting from raw means, standard errors, p-values, t-statistics, or confidence intervals), binary outcomes (converting from risk differences, odds ratios, or linear probability model coefficients), and clustered designs (applying design effect corrections when authors have not already adjusted for clustering). Each conversion is documented with its formula, inputs, and quality checks, and flagged if the resulting effect size falls outside plausible ranges.

Phase 4: Automated Validation

As a final quality assurance step, we run every extracted estimate back through an independent validation stage. Each paper's full text is re-read alongside its extracted entries, and a separate LLM evaluates whether each estimate, including the effect size, confidence interval, treatment arm linkage, outcome description, and causal interpretation, is consistent with what the source document actually reports. Each entry receives a consistency score and targeted feedback identifying any discrepancies. Estimates that fall below quality thresholds are flagged for manual review, ensuring that the database maintains high accuracy even as it scales to thousands of papers.

Expert-Validated Reliability

The pipeline's reliability has been validated against expert human annotations. Seven trained annotators, each holding at least a Master's degree in economics with experience in RCT research, produced gold-standard extractions following IDEAL consortium annotation guidelines. Against these benchmarks, the pipeline achieves precision levels that closely approach expert performance for identifying the correct intervention, outcome, and associated effect size.

The system is most reliable on papers with standard reporting conventions and well-structured tables, which account for the majority of published RCTs. It becomes more challenging for complex multi-arm factorial designs with many treatment arms, papers reporting only regression coefficients without sufficient descriptive statistics, and very long tables spanning many pages.

The ImpactAI RCT Database

We have applied this pipeline across a wide range of sources, including papers from four peer-reviewed economics journals (QJE, AEJ: Applied, JDE, and World Bank Economic Review), three major impact evaluation repositories (J-PAL, IPA, 3ie), and additional papers indexed in NBER, SSRN, CEPR, Scopus, Semantic Scholar, EconLit, and other academic databases. The result is one of the largest structured databases of experimental evidence in development economics.

Each estimate in the database is a richly structured record that captures the full experimental context of a finding. Far beyond a simple effect size number, every estimate encodes the complete chain from intervention to outcome, with detailed metadata at each level.

This structured representation powers ImpactAI's ability to answer complex evidence questions in seconds: comparing the effectiveness of different interventions on a given outcome, identifying evidence gaps across sectors and geographies, and performing real-time meta-analyses that would traditionally take weeks or months of researcher time.

The extraction pipeline and the underlying schema have also been formalized in our research on Hierarchical Information Extraction (HIE), the task of reconstructing the complete multi-level experimental design of an RCT from its full-text document. The database is continuously growing as new studies are published and processed.