A controlled study of whether large language model "agents" can turn messy hospital lab and medication records into a research ready format, while keeping every clinical fact intact.
Every hospital records data its own way. To pool it for research, that data must be reshaped into a shared standard, and the danger is subtle: structural checks pass while clinical meaning quietly breaks.
A widely used "common data model": shared tables and rules so different hospitals' data can be analysed together.
Extract, Transform, Load. The pipeline that pulls data from a source system, reshapes it, and loads it into the target.
Reshaping records into a shared model is not just renaming columns; it is a meaning preservation task. A lab result with the wrong unit, a local code mapped to the wrong standard, or a medication order mistaken for an administration each passes a structural check yet corrupts the science.
That is why this study targets lab results and medications: their meaning is spread across many fields at once (code, unit, value, dose, route, timing), so they are the highest stakes cases to get right. The blunt takeaway: "the pipeline ran" is not the same as "the data are correct."
Glucose 65.5 mg/dL
3.635213 mmol/L
"2.5 mg QD"
"2,5 mg ORAL daily TABLET"
Metformin ER (extended-release)
a less-specific drug code
Judging the AI fairly needs a known-correct answer for every record, impossible with real patients. So the benchmark is fully synthetic and privacy safe: the right answer is fixed in advance, then the source data is made messier in controlled steps.
Synthetic patient records. No real people, fully shareable.
Add the lab and medication detail a real benchmark needs.
Fix the exact correct target answer to grade against.
Layer in realistic noise: codes, units, dates, formats.
Grade each AI automatically, field by field.
Because the answer key is known and the data synthetic, the whole pipeline is open, auditable, and reproducible. A deliberate methodological choice, not just a privacy workaround.
The source backbone is a Synthea-derived cohort (seed 42): 52 patients, 390 encounters, 1,222 lab rows, 213 medication orders, 6,450 medication administrations. Raw exports become five canonical tables, then deterministic OMOP CDM v5.4 truth for the measurement and drug_exposure targets (person and visit_occurrence act as controls).
Ground-truth artifacts (schema map, term map, transform map, attainable mask, provenance manifests) ship with the data so scoring never depends on after-the-fact human judgement. Lab rows retain LOINC truth; medications retain RxNorm truth, via deterministic local-code maps.
Real hospital data is rarely random noise; it is locally coherent but externally confusing. Five families of variation stack one on the next, so each agent faces a steadily harder version of the same records. Every layer stays recoverable, so the test rewards skill, not luck.
Out of scope by design: broken record linkage, pipeline glitches, and genuinely ambiguous data. These need human adjudication and are flagged as future work.
The leader's question: does more tooling and autonomy actually make the AI better? Three designs add exactly one capability at a time.
Reads the schema and sample rows, then writes a mapping plan from its own knowledge. No lookups, no tools.
Everything A does, plus the ability to look up official medical codes for lab tests and drugs as it works.
A full workspace agent: it inspects files, writes and runs code, validates its own output, fixes mistakes, finalises.
A emits a mapping spec applied by a deterministic executor (tests model-internal knowledge alone). B keeps A's prompt-to-spec and executor contract but adds UMLS/LOINC and RxNorm/RxNav tool paths, isolating terminology grounding as the only added variable. C shifts to a sandboxed single-agent workspace (shell, Python, local terminology tools, validation and finalisation) on a matched local-tool profile with web search disabled, so it gets no browsing advantage over A and B.
More capability did help: the workspace agent (C) scored highest or tied-highest on both targets. But the gains were uneven, and came with operational costs a single accuracy score would hide.
Accuracy went up with capability. The workspace agent led on both data types.
Reliability went down. The simplest agent (A) never failed; tool-heavy agents hit rate limits and policy errors that aborted runs.
Cost climbed steeply for a roughly 1 to 2 point medication gain: most tools, tokens, and time.
Two models, near-tie on quality. GPT-5.4 and Kimi-K2.6 scored almost identically, but GPT-5.4 was more reliable and far more efficient.
The honest read: "more agentic" is not automatically "better." It depends on data type, reliability needs, and budget.
Accepted (credits deterministic exceptions plus reviewed unit/value equivalence) vs exact (strict surface): measurement exact about 96 to 97% while accepted hit ceiling; drug exact about 90.7%. Pairwise bootstrap at full L1 to L5: C beat A on accepted drug_exposure by 1.25 pp (GPT, CI 0.64 to 1.86) and 1.96 pp (Kimi, CI 0.24 to 4.81); C vs B was only 0.24 pp for GPT (CI -0.57 to 1.25), a small layer-dependent edge, not a clean ordering.
Robustness: from L1 to the full stack, accepted-score retention stayed 0.992 to 1.012. Cumulative noise mostly changed which errors appeared rather than collapsing the score.
Reliability and cost: Agent A had zero failures; failures concentrated in tool-heavy B and C. A zero-imputation check (failures scored 0) flips Kimi's ordering toward A. Agent C tokens per source-row: Kimi 11,639 vs GPT 3,986 (drug), 1,644 vs 377 (measurement); average runtime 506.6s vs 137.1s.
A single accuracy number hides clinically different mistakes. Breaking errors down by type is the study's most decision-relevant finding.
Lookup tools cut likely-wrong code mappings sharply, from about 32% down to single digits.
The oversight lesson: medication data needs closer human review than lab data, even at high headline accuracy. A plausible-looking medication table is not proof it is correct.
"The pipeline ran" is not the same as "the data are correct." Quality must be measured at the level of clinical meaning.
Before any patient-data use: source-quality controls, deterministic validation, governance review, and expert oversight remain non-negotiable.