Long read· Master's thesis · Jun 2026
Can AI agents standardise clinical data without losing its meaning?
A controlled test of whether AI agents can turn messy hospital lab and medication records into a research-ready format and keep every clinical fact intact.
Clinical fields kept their meaning, synthetic data
- Author
- Arvind Guruprasad
- Supervisor
- Vladimir Li, Arkus AI
- Models
- GPT-5.4 and Kimi-K2.6
- Programme
- MSc, KI, KTH and SU
The bottom line
Yes, mostly. The best agent kept lab results almost perfectly and medications about 91% of the time. Not yet for real patients: this is a lab benchmark on synthetic data, and human oversight stays essential.
- 3agent designs compared
- 2models benchmarked
- 5stacked layers of messiness
- ~91%medication accuracy, best agent
01The stakes
A converted table can look valid and still be clinically wrong.
Every hospital records data its own way. To pool it for research, the data has to be reshaped into a shared standard, and the risk is quiet: structural checks pass while the clinical meaning breaks.
- OMOP
- A common data model: shared tables and rules so different hospitals' data can be analysed together.
- ETL
- Extract, transform, load. The pipeline that pulls data out, reshapes it and loads it into the target.
Lab results and medications carry their meaning across many fields at once: code, unit, value, dose, route and timing. That makes them the cases where getting it wrong costs the most.
- Safe
Lab unit conversion
Glucose 65.5 mg/dL3.635213 mmol/LDifferent text, same clinical value.
- Ambiguous
Medication instructions
"2.5 mg QD""2,5 mg ORAL daily TABLET"Still readable, harder to compare.
- Risky
Drug specificity
Metformin ERgeneric metformin codeThe exact formulation is no longer traceable.
02The test
A known answer for every record, so the grading is fair.
Grading an agent needs the correct answer for every record, which real patients can't give you. So the benchmark is fully synthetic: the right answer is fixed first, then the source data is made messier in controlled steps.
- Generatesynthetic patients
- Enrichlab and drug detail
- Lock the truthfixed answer key
- Add messinessfive layers
- Scorefield by field
Technical detailThe dataset
A Synthea-derived cohort (seed 42): 52 patients, 390 encounters, 1,222 lab rows, 213 medication orders and 6,450 medication administrations. Raw exports become five canonical tables, then deterministic OMOP CDM v5.4 truth for the measurement and drug_exposure targets, with person and visit_occurrence as controls.
Ground-truth artifacts (schema map, term map, transform map, attainable mask, provenance manifests) ship with the data, so scoring never depends on human judgement after the fact. Labs keep LOINC truth and medications keep RxNorm truth.
03Messiness
Five layers of real-world mess, stacked one on the next.
Hospital data is rarely random noise. It is locally sensible and confusing to everyone else. Each layer adds one family of variation, so every agent faces a harder version of the same records, and every layer stays recoverable.
- L1
Representation
Decimal commas, local null markers, odd date formats, local category codes.
- L2
Schema and contract
Renamed fields, timestamps split apart, dose and reference text merged.
- L3
Terminology
Local lab codes, formulary IDs and brand names instead of standard codes.
- L4
Units and values
Unit swaps and scale changes: the right concept at a different magnitude.
- L5
Timing
Workflow timestamps shifted or redacted: which date is the real event?
Left out on purpose: broken record linkage, pipeline glitches and genuinely ambiguous data. Those need a human to decide.
04The agents
Three designs, each given one more capability.
The question a team lead would ask: does more tooling and autonomy make the agent better? Each design adds exactly one thing.
A
Reasoning only
Reads the schema and sample rows, then writes a mapping plan from its own knowledge.
- Schema reasoning
B
With code lookups
Everything A does, plus looking up official lab and drug codes as it works.
- Schema reasoning
- Code lookups
CBest overall
Full workspace
Inspects files, writes and runs code, checks its own output and fixes mistakes.
- Schema reasoning
- Code lookups
- Runs code, self-checks
Technical detailArchitecture contracts
A emits a mapping spec applied by a deterministic executor, which tests model knowledge alone. B keeps that contract and adds UMLS/LOINC and RxNorm/RxNav tools, so code lookup is the only new variable. C is a sandboxed single-agent workspace (shell, Python, local terminology tools, validation) with web search off, so it gets no browsing advantage.
05Results
More capability helped, unevenly, and at a cost.
The workspace agent scored highest or tied on both targets. A single accuracy number hides what it cost to get there.
- Accuracy rose with capability. The workspace agent led on both data types.
- Reliability fell. The simplest agent never failed; tool-heavy agents hit rate limits and policy errors.
- Cost climbed steeply for a one-to-two point gain on medications.
- The models tied on quality, but GPT-5.4 was more reliable and far cheaper to run.
The honest readMore agentic doesn't automatically mean better. It depends on the data, how reliable it has to be, and the budget.
Technical detailConfidence intervals, robustness and cost
Accepted (credits deterministic exceptions and reviewed unit equivalence) vs exact match: measurement exact was about 96–97% while accepted hit the ceiling; drug exact was about 90.7%. At full L1–L5, C beat A on accepted drug_exposure by 1.25 pp (GPT, CI 0.64–1.86) and 1.96 pp (Kimi, CI 0.24–4.81). C vs B was only 0.24 pp for GPT (CI −0.57–1.25).
From L1 to the full stack, accepted-score retention stayed between 0.992 and 1.012: noise changed which errors appeared more than the score. Agent C used 11,639 tokens per source row on Kimi vs 3,986 on GPT for drugs, with average runtime 506.6 s vs 137.1 s.
06Failure modes
Labs and medications fail in different ways.
Breaking errors down by type is the most useful finding for anyone deciding where to put human review.
Lab results · mostly cosmetic
- UnitConversions that change the text, not the clinical value.
- AuditLabels swapped for codes: right answer, weaker trail for a human.
Code lookups cut likely-wrong mappings from about 32% to single digits.
Medications · consequential
- DoseInstructions reworded so they no longer compare cleanly.
- DrugProduct specificity lost, like extended-release mapped to a generic code.
- RouteRoute of administration dropped on some rows.
- TimeExposure dates shifted by a day.
07So what
“The pipeline ran” doesn’t mean the data is correct.
Quality has to be measured at the level of clinical meaning. What that means depends on who is deciding.
Clinical leaders
- Promising, not deployable: validated only on synthetic data with a known answer key.
- Keep a human on medications: dose, route, product and timing are where meaning slips.
- Ask for semantic checks and traceability you can audit, not just a successful load.
Technology leaders
- More autonomy added cost, latency and failures for a small gain.
- Treat reliability as a metric. The simplest design never failed.
- Two strong models tied on quality but differed up to about three times on cost and runtime.
A reproducible way to test clinical AI before it touches real patients.
The generator, answer key, messiness layers, agents and scoring are all open source and run on synthetic data, so anyone can inspect, re-run or extend it.
What this doesn't prove
- Synthetic data in a controlled benchmark, not a deployment.
- Only lab results and medications.
- Only messiness that can be recovered; truly ambiguous records are future work.
- Real use would need governed clinical data, approvals, security review and expert oversight.