AI-Generated Clinical Notes Pose Risks of Dangerous Omissions and Fabrications

Sebastian Fox detailed a case where an AI-generated clinical note described a patient's headache as routine, suggesting paracetamol and follow-up if symptoms persisted. This seemingly benign note, however, dangerously omitted a crucial symptom: jaw pain, which is indicative of giant cell arthritis.

Untreated, giant cell arthritis can lead to irreversible vision loss. The AI note was technically coherent but failed to capture a critical diagnostic symptom, illustrating how AI can create "invisible dangers" through omissions that appear routine.

In another real-world instance, a young patient, in their twenties, was treated for tonsillitis. The AI model documenting the encounter fabricated significant medical details, including chest pain, a non-existent diabetes diagnosis, and medications the patient had never taken, alongside an address for a hospital that did not exist.

This fabrication led to serious downstream consequences, as the patient was later incorrectly summoned for diabetic eye screening based on the erroneous AI-generated record.

A large-scale real-world study of AI-generated clinical notes revealed that approximately 1 in 20 notes contained an error serious enough to potentially cause significant harm to a patient. Beyond critical errors, 1 in 5 notes were found to have an important omission, meaning vital information was left out.

Furthermore, 1 in 10 notes included a hallucination, where the AI fabricated information not present in the original interaction. These statistics underscore the significant safety risks posed by current AI systems in healthcare documentation.

Currently, there is a lack of standardized adverse event reporting mechanisms specifically for AI-generated clinical notes, making it challenging to systematically track and address these issues across the healthcare system.

Testing three leading production Ambient Scribes on the market demonstrated that automated detection systems frequently fail to identify dangerous errors. Sebastian Fox explained that most errors, particularly those with high stakes, often fall below the threshold of automated detection.

The most critical failures are high-stakes omissions that appear entirely coherent within the note, making them difficult for automated systems to flag as incorrect or missing.

AI models can make subtle inference errors that significantly alter clinical decision-making. In one instance, a patient simply stated their headache 'just happened,' but the AI recorded this as 'abrupt sudden onset.'

Abrupt onset is a critical red flag suggesting a brain bleed, which would change the entire clinical workup and potentially lead to unnecessary or incorrect interventions. In another case, the AI ignored a doctor and patient's agreed plan to try antibiotics, instead documenting a plan for additional tests, demonstrating a fundamental misunderstanding of the clinical context.

Sebastian Fox outlined that AI errors in clinical documentation stem from two main sources: transcription inaccuracies and over-inference during the note generation phase. Transcription errors include mishearing similar-sounding terms, such as "Humalog" for "Humulin" or "hyperthyroidism" for "hypothyroidism," which can lead to critical medication or diagnosis errors.

Generation errors, on the other hand, can occur even with a perfect transcript. These errors involve the AI either adding information that was never said, changing information that was spoken, or omitting crucial details that should have been included in the note.

The fundamental challenge lies in differentiating helpful rephrasing or summarization by the AI from dangerous fabrications or omissions. This distinction is crucial for patient safety.

The speaker emphasized that the AI's ability to 'hallucinate' or invent details, even when given accurate input, remains a significant hurdle in ensuring the reliability and safety of automated clinical documentation.

To effectively evaluate AI in clinical settings, Sebastian Fox proposed a three-step repeating loop framework. This framework begins with discovering actual failure modes from real system outputs, rather than relying on hypothetical scenarios.

The second step involves capturing how human experts judge these outputs, focusing on their reasoning and corrections. Finally, the system's output is calibrated against these expert judgments, with the understanding that the "standard" of correctness is dynamic and evolves, requiring the evaluation loop to move with it.

Sebastian Fox stressed that evaluation rubrics should not be created in isolation but must be derived from actual system performance. He argued that writing a rubric "in a vacuum" is ineffective because real-world system failure modes are essentially unbounded and unpredictable.

To properly identify what goes wrong, AI systems must be deployed in production environments, and their actual outputs must be closely examined. By clustering similar errors, the true failure modes naturally surface, revealing problems that synthetic test cases, which only catch previously imagined errors, would miss.

When capturing expert judgments, the focus should extend beyond mere scores to include the reasoning and corrections provided by human experts. Sebastian Fox explained that this qualitative feedback is invaluable for understanding the nuances of AI performance.

Over time, collecting these detailed reasons and corrections builds a comprehensive record of how experts actually evaluate the system. This process does not require excessively long review sessions to be effective, allowing for efficient integration into workflows.

Sebastian Fox recommended assembling evaluation standards dynamically, "on the fly," rather than relying on static rubrics. For each AI output, a judging agent should pull in all relevant context specific to that case. This includes its memory of similar outputs that have been previously judged and scored, applicable expert corrections, and relevant reference documents or guidelines.

This approach avoids the limitations of static rubrics, which quickly become outdated, and reduces the need for frequent and costly model retraining. By using dynamic references, the evaluation system can adapt to evolving standards and provide more accurate assessments of AI performance.

Sebastian Fox illustrated how providing the right context improves diagnostic accuracy with a clinical example. He referred to the initial case of a patient with a severe headache misclassified as routine due to a dropped red flag.

The AI model itself is capable, but it needs appropriate context to reason correctly, not merely to memorize cases. When the model is given the relevant information and held against specific clinical standards, the dropped red flag becomes obvious. This suggests that catching such errors is not inherently difficult; the AI simply lacked the understanding of what information was critical in the given context.

Sebastian Fox presented comparative data on evaluation system performance, highlighting that strong off-the-shelf "judges" often miss crucial details because they lack the specific, evolving context of clinical standards. Static rubrics, while initially helpful, quickly become stale as medical understanding and best practices shift over time, rendering them ineffective for continuous improvement.

In contrast, the dynamic, loop-based evaluation system, which continuously discovers failure modes and calibrates outputs against expert judgments, demonstrated significantly superior performance on the specific dataset. This approach allows for ongoing learning and adaptation, ensuring that the AI remains aligned with current clinical realities and expert expectations.

The continuous learning inherent in the dynamic loop model allows it to adapt to evolving standards, leading to more reliable and safer AI-generated clinical documentation over time.

Sebastian Fox concluded by emphasizing that AI evaluation cannot be a one-time build-and-freeze activity. He stated that the true standard of 'good' in clinical AI does not exist as a fixed document but must be continuously discovered from real-world outputs and validated by the experts who hold that knowledge.

This necessitates the collection of raw material through free-form expert comments and continuous engagement to keep the evaluation process alive and adaptive as standards evolve. The goal is to build a system that learns and adjusts, reflecting the dynamic nature of medical practice.