How to Evaluate AI Accuracy in the Systematic Literature Review Process?

SLR and AI Accuracy

Evaluating AI Accuracy in the SLR Process requires measuring performance against dual-human reference standards on title and abstract screening, full-text eligibility, and structured data extraction using recall, precision, specificity, and overall accuracy, while enforcing human validation of every consequential decision. Real-world multi-project data from regulatory literature reviews show that structured AI-augmented frameworks achieve consistent accuracy in the 84–90 percent range when humans retain final accountability.

Life sciences teams preparing Clinical Evaluation Reports, Periodic Safety Update Reports, health technology assessment dossiers, and other submission-grade evidence face rising publication volumes and fixed timelines. Traditional manual workflows struggle with reviewer fatigue, inconsistency, and limited scalability. Artificial intelligence can reduce effort on high-volume steps, yet only rigorous evaluation protects methodological integrity and regulatory defensibility.

Key Takeaways

Dual-Human Reference Standards: AI accuracy must be measured against adjudicated dual-human decisions rather than single-reviewer passes.

• Stage-Specific Metrics: Prioritize recall and precision for screening, field-level accuracy for extraction, and Cohen’s kappa for risk of bias.
• Human-in-the-Loop Governance: Regulatory frameworks achieve 84–90% accuracy and significant time savings only when human reviewers maintain final accountability.
• Continuous Monitoring: Utilize decision-level audit logs and reviewer override rates to track performance continuously post-launch.

This article serves HEOR, market access, medical affairs, and clinical development leaders who must decide whether an AI-assisted systematic literature review (SLR) will withstand scrutiny from a health technology assessment (HTA) body. It explains which metrics to request, how to build a reference standard, how many records a test needs, and how to keep measuring after go-live.

What Metrics Define Reliable AI Accuracy?

Class imbalance is severe in most literature searches, so overall accuracy alone can mislead. Prioritize these metrics calculated against a dual-human adjudicated reference standard:

• Recall: the proportion of truly relevant records correctly identified. High recall is essential to avoid missing evidence.
• Precision: the proportion of records flagged as relevant that are truly relevant.
• Specificity: the proportion of irrelevant records correctly excluded.
• Overall accuracy and F1 score: useful summary measures when interpreted alongside the others.

Independent systematic evaluations of automated screening report combined recall near 0.93 when models are optimized for maximized recall. Data-extraction studies show overall accuracy ranging widely by data type, with categorical fields extracted more reliably than numerical effect sizes. Omissions dominate error patterns, while hallucination rates remain low under controlled conditions.

These external benchmarks provide context. Internal validation on the specific protocol and corpus remains indispensable.

Which Metrics Measure AI Accuracy at Each SLR Stage?

Each SLR stage needs its own metric because each stage fails in a different way. Screening fails by omission, extraction fails by wrong or missing values, and appraisal fails by inconsistent judgment.

Table 1: Accuracy Metrics by SLR Stage

How to Build a Reference Standard That Holds Up

A defensible reference standard comes from adjudicated human decisions, not from one reviewer’s first pass. Human screening carries its own error: Wang et al. (2020) measured a 10.76 percent error rate for individual reviewers at abstract screening. Consequently, a test that treats every human decision as correct penalizes the tool for human mistakes.

Five Steps to Test AI Accuracy in the SLR Process

1. Select a completed review. Choose one in your own therapeutic area with documented screening decisions and extraction tables.
2. Freeze the configuration. Record the model version, prompts, and thresholds before the test starts.
3. Run the tool blind. Keep the original human decisions hidden from the tool and from the operator.
4. Adjudicate every disagreement. Assign a senior reviewer who sees both decisions without labels.
5. Report by stage. Publish each metric with a 95 percent confidence interval and attach the full error list.

How Many Records Does a Valid Test Need?

Sample size limits what a test can prove. The rule of three, a standard statistical shortcut, states that when a test observes zero failures in n cases, the 95 percent upper bound on the true failure rate is about 3 divided by n. A tool that misses none of 30 included studies has therefore shown only that recall is at least 90 percent. A lower bound of 95 percent requires about 60 included studies with no misses, and a lower bound of 99 percent requires about 300. When a single review includes fewer than 60 studies, teams should pool several completed reviews before they accept a recall claim.

Where Do Accuracy Checks Sit in an AI-Assisted SLR Workflow?

Accuracy checks belong at every handoff between the model and the reviewer, not only at the end. A measurable workflow has six checkpoints:

1. Ingestion. The system parses each PDF and logs the tables and figures it could not read.
2. AI suggestion. The model returns a decision, a confidence score, and a stated reason.
3. Human review. A named reviewer accepts, edits, or rejects each suggestion.
4. Adjudication. A second reviewer resolves conflicts between the model and the first reviewer.
5. Audit log. The system stores the model version, prompt, decision, reviewer, and timestamp.
6. Monitoring. The team recalculates each metric from the log at every update.

In this design, the reviewer edit rate becomes a continuous accuracy signal. Buyers of generative AI life sciences platforms should confirm that the audit log exports at the decision level, because a document-level change history cannot reproduce these metrics.

What Does AI Accuracy Look Like in Real Regulatory Reviews?

Real projects show accuracy as a range that holds across volumes, not as a single benchmark score. At ISPOR 2026, MadeAi presented a multi-project analysis of five regulatory literature reviews that supported Clinical Evaluation Reports and Periodic Safety Update Reports. One fully manual project served as the baseline. Four projects used an AI-augmented workflow in which human reviewers kept responsibility for adjudication, validation, and quality oversight at every stage.

Table 2: AI Accuracy and Effort Across Five Regulatory Literature Reviews

Three points matter for evaluation. First, the study reports AI accuracy of 84 to 90 percent across screening and data extraction while record volume grew about 19-fold, which indicates stable performance at scale. Second, the team measured accuracy inside live workflows with human validation, not on a curated benchmark. Third, the figures leave questions that every buyer should still ask: the recall behind each screening result, the split between numeric and categorical fields, and the confidence interval for each project. The article on AI-augmented literature reviews describes the workflow in full.

Key Features that Make Accuracy Measurable

Accuracy is measurable only when the platform exposes the evidence behind each output. Look for these features:

• Confidence scores and stated reasons for every screening decision
• Source highlighting that links each extracted value to its passage
• Version pinning for models and prompts
• Configurable recall thresholds
• Decision-level audit exports
• Reviewer override tracking

Who Benefits from Rigorous Accuracy Evaluation?

Rigorous evaluation benefits every team that submits or acts on synthesized evidence, because it converts a vendor claim into a documented methods statement.

Key Beneficiaries of AI Accuracy Evaluation

• HEOR. Teams that adopt HEOR evidence synthesis with AI can cite stage-level metrics in the methods section of a network meta-analysis or an economic model.
• Market access. Dossier teams can answer assessor questions on AI use with a validation report. The guide to choosing a JCA dossier preparation platform explains the disclosure requirements.
• Clinical development. Teams responsible for clinical operations in pharma rely on literature reviews for protocol design, endpoint selection, and safety surveillance, where a missed study can affect trial decisions.
• Outsourced delivery. Buyers of systematic literature review services can write metric thresholds into each statement of work.

Best Practices for Ongoing Accuracy Monitoring

Accuracy is a property of a specific model version on a specific task, so evaluation must continue after go-live. First, revalidate after every model or prompt change. Second, sample a fixed share of AI exclusions for human re-review in every project. Third, track the reviewer edit rate by field type and investigate each increase. Fourth, set the recall threshold in the protocol before screening starts. Finally, report AI use with the tool name, version, date, and validation results.

Risks and Limitations of Accuracy Evaluation

Evaluation reduces risk, but it does not remove it. Five limitations deserve attention:

• Benchmark contamination. Public test articles may sit in model training data, which can inflate results.
• Small samples. A 10-trial extraction study indicates promise, not production performance.
• Limited transfer. Results from one therapeutic area or study design do not predict another.
• Imperfect reference standards. Human error sets a ceiling on measurable accuracy.
• Over-trust. A strong validation report can lead reviewers to accept suggestions without reading them.

Clark et al. (2025) concluded that current evidence does not support generative AI use in evidence synthesis without human oversight.

What Comes Next for AI Accuracy Evaluation?

Reporting standards are converging, and assessors will expect structured validation evidence. The RAISE recommendations, which the Cochrane-led position statement supports, continue to evolve (Flemyng et al., 2025). In addition, the HTACG principles make AI disclosure part of every JCA dossier, including the tool name, version, date, developer, and purpose (HTACG, 2026). Multi-project analyses of live workflows will supplement single benchmark studies. Expect continuous monitoring from audit logs to replace one-time validation.

Conclusion

Evaluating AI accuracy in the SLR process is a measurement exercise, not a matter of trust. Define the metric for each stage, build an adjudicated reference standard, test enough records to support the claim, and keep measuring from the audit log. Teams that follow this sequence can state, with evidence, how their AI-assisted review performs. As a practical next step, rerun one completed review through any shortlisted tool and compare the stage-level results with your own records.

Author’s Note: This article was supported by AI-based research and writing, with Claude 5.1 assisting in the creation of text and images.