Validation research and evidence
Separate retrieval, evidence selection, answer grounding and clinical review. These research methods inform experiment design; they are not product contracts or release acceptance.Claims need evidence
Record-level evidence and reviewer rationale matter more than aggregate scores. Test doubles establish runner mechanics, not clinical safety. Keep final answers, In-Depth material and rejected drafts distinct, and review/redact artifacts before publication.
Evaluation methodology
| Area | Measures | Why it matters |
|---|---|---|
| Retrieval | Precision@k, recall/coverage@k, miss@k, empty-answer correctness, latency | Medical RAG studies show retrieval is often the main bottleneck; one large expert eval found only about 22% of top-16 passages relevant. |
| Test-data fidelity | Import success, schema compatibility, concept mapping coverage, serialized resource coverage, known-answer fixtures | OpenMRS demo data is version-sensitive; modern Ref App validation needs an importable corpus whose observations, concepts, encounters, notes, and demographics survive indexing. |
| Evidence selection | Which retrieved records the answer actually cites or uses | Separate retrieval quality from LLM use of evidence; relevant records can be retrieved and still ignored. |
| Answer grounding | Claim-level support against retrieved context; unsupported, partially supported, contradicted | Use RAGAS/TruLens-style faithfulness as a screening metric, but require clinician adjudication for acceptance thresholds. |
| Citation quality | Statement-level citation precision and recall | MedCite-style evaluation is a better fit than answer-level citation counts; every clinical claim should map to records. |
| Abstention | Correct no-record/no-answer behavior, especially for negative clinical questions | FHIR-AgentBench keeps empty-answer cases; OpenMRS should make absent-data cases first-class, not edge cases. |
| Security | Prompt injection, system prompt leakage, schema escape, sensitive data disclosure | OWASP LLM Top 10 2025 maps directly to regression suites plus structured-output validation. |
| Clinical review | Blinded clinician scoring, inter-rater agreement, adjudication, reviewer instructions | TRIPOD-LLM asks for assessor qualifications, annotation guidelines, and agreement reporting. |
| Governance | Dataset transparency, intended use, change protocol, monitoring, rollback criteria | NIST, CHAI, FDA PCCP, and STANDING Together all point toward documented lifecycle controls. |
Research sources
| Source | Finding | Use in experiment design |
|---|---|---|
| OpenMRS Demo Data | Documents platform-versioned demo SQL datasets, including large demo data, plus the modern Ref App generator path. | The dated demo-data profile describes a transformed 2.8-compatible corpus. Record dataset provenance and verify import fidelity for the version used in an experiment. |
| openmrs-module-referencedemodata | Reference Application module that creates demo patients on startup from the global property path documented by OpenMRS. | Use this as a current Ref App schema/metadata reference and control data path, not as the replacement for the large demo corpus. |
| openmrs-content-referenceapplication-demo | Reference Application demo content package with starter metadata such as labs, diagnoses, drugs, and configuration. | Use as the metadata side of the mapping, especially when legacy SQL concepts do not align with current Ref App concepts. |
| Rethinking RAG for Medicine | Stage-aware expert evaluation: retrieval, evidence selection, response factuality/completeness. | Use its decomposition as the core OpenMRS validation model. |
| VeriFact | EHR-grounded fact verification via RAG plus LLM-as-judge, compared with clinician ground truth. | Adapt claim-level support labels for chart answers. |
| MedCite | Medical citation generation and evaluation with statement-level citation precision and recall. | Use citation precision/recall, not citation count, for answer quality. |
| FHIR-AgentBench | 2,931 clinical questions grounded in FHIR; reports retrieval and answer correctness, including empty answers. | Borrow benchmark structure for querystore and OpenMRS resource-level QA. |
| MedHELM | Holistic medical LLM evaluation emphasizing realistic tasks, safety, and reproducibility. | Use for model-selection discipline, not as a replacement for OpenMRS-specific evals. |
| TRIPOD-LLM | Healthcare LLM reporting checklist with data, metrics, annotation, prompting, compute, and intended-use items. | Turn into metadata requirements for every eval run. |
| NIST GenAI Profile | Risk-management lifecycle for design, development, use, and evaluation of GenAI systems. | Use as governance backbone for release gates and monitoring. |
| CHAI RAIC | Healthcare AI self-review across usefulness, fairness, safety, transparency, privacy/security. | Use for clinical-readiness checklist and review packet. |
| STANDING Together | Dataset diversity, inclusivity, generalisability, and transparency recommendations. | Track who is represented in eval patients and clinical question sets. |
| OWASP LLM Top 10 2025 | Security risk taxonomy for prompt injection, leakage, vector weaknesses, misinformation, and more. | Expand PromptInjectionEvalTest into a maintained red-team corpus. |
| RAGAS Faithfulness | Claim-support scoring against retrieved context. | Good automated screen; do not use as sole clinical acceptance criterion. |
| TruLens RAG Triad | Context relevance, groundedness, and answer relevance. | Matches the desired OpenMRS split: retrieve right records, cite facts, answer the question. |