Diagnostic Accuracy of GPT-4o and Claude 4.6 Sonnet in Turkish ED Anamnesis Notes
Trial statusRecruiting
Trial phaseNot listed
Trial typeObservational
Biological sexAll
Age18+
SponsorMarmara University Pendik Training and Research Hospital
About this trial
This retrospective diagnostic accuracy study evaluates the ability of two large language models (LLMs) - GPT-4o (gpt-4o-2024-11-20; OpenAI) and Claude 4.6 Sonnet (claude-sonnet-4-6; Anthropic) - to generate correct diagnoses from anonymized Turkish-language emergency department (ED) anamnesis notes, and compares their performance with the diagnosis entered by the treating emergency physician. A consensus gold standard is established by three independent board-certified emergency medicine specialists who blindly review each note and vote on the primary diagnosis using ICD-10 three-character codes; the majority vote (at least 2 of 3 specialists agreeing) constitutes the reference standard. Both LLMs are evaluated using a standardized zero-shot direct prompting strategy (temperature=0, stateless API sessions). The primary outcome is diagnostic accuracy (proportion of ICD-10 chapter-level matches) and Cohen's kappa for each LLM against the gold standard. Secondary outcomes include top-3 accuracy, treating physician accuracy, inter-model agreement, and subgroup analyses by ESI triage level and ICD-10 chapter. Inter-rater reliability among the three specialists is quantified using Fleiss' kappa. Analyses are performed in Jamovi. This study represents the first evaluation of LLM diagnostic accuracy using Turkish-language clinical notes and the first to benchmark LLM performance against an independent three-specialist majority-vote gold standard rather than against the treating physician's own diagnosis.
Eligibility criteria
This trial does not accept healthy volunteersQualifiers
Adult patients (aged 18 years and older) presenting to the emergency department.
Complete electronic health record available in the hospital information system (HBYS) containing a detailed anamnesis note with chief complaint, symptom duration, associated symptoms, and relevant medical history.
A definitive primary diagnosis recorded by the treating emergency physician using ICD-10 codes at the time of patient file closure.
Disqualifiers
Emergency department anamnesis notes containing fewer than 50 words or completely lacking substantive clinical content[cite: 1].
Pediatric cases (age under 18 years)[cite: 1].
Patients critically ill and triaged to high-acuity resuscitation areas (Emergency Severity Index [ESI] level 1)[cite: 1].
Clinical notes containing residual identifying information that cannot be fully de-identified, preventing compliance with data privacy regulations[cite: 1].
Trial population
The study population comprises consecutive adult patients (aged 18 years and older) who presented to the emergency department of a tertiary care training and research hospital and had their encounters fully documented in the hospital information system (HBYS). Eligible individuals must have a complete electronic anamnesis note containing the chief complaint, history of present illness, and clinical presentation, alongside a definitive primary ICD-10 diagnosis finalized by the treating emergency physician at file closure. The population excludes pediatric cases, patients triaged to high-acuity resuscitation areas (ESI level 1), and clinical notes with fewer than 50 words or insufficient clinical content.
Trial design
Design model
Cohort
Time perspective
Retrospective
Treatments tested in this trial
Not listed
Trial groups
600 Participants
are grouped into 1 trial groupGroup A: Emergency Department Patient Cohort
Trial outcomes
Primary outcomes
1
Diagnostic Accuracy of GPT-4o for ICD-10 Chapter-Level Diagnosis
Proportion of cases in which GPT-4o primary (rank 1) diagnosis matches the 3-specialist majority-vote gold standard at the ICD-10 chapter level (22 categories). Range: 0 to 1.00.
Time frame
At the time of single-session algorithmic evaluation (each case evaluated once following data extraction in June 2026).2
Diagnostic Accuracy of Claude 4.6 Sonnet for ICD-10 Chapter-Level Diagnosis
Proportion of cases in which Claude 4.6 Sonnet primary (rank 1) diagnosis matches the 3-specialist majority-vote gold standard at the ICD-10 chapter level (22 categories). Range: 0 to 1.00.
Time frame
At the time of single-session algorithmic evaluation (each case evaluated once following data extraction in June 2026).Secondary outcomes
1
Cohen's Kappa Between GPT-4o Primary Diagnosis and Gold Standard
Kappa coefficient measuring agreement between GPT-4o rank-1 ICD-10 chapter and the 3-specialist gold standard . Interpreted per Landis \& Koch (1977): \<=0.20 slight; 0.21-0.40 fair; 0.41-0.60 moderate; 0.61-0.80 substantial; \>0.80 almost perfect . Range: -1.00 to 1.00 .
Time frame
At the time of algorithmic evaluation (June-July 2026)2
Cohen's Kappa Between Claude 4.6 Sonnet Primary Diagnosis and Gold Standard
appa coefficient measuring agreement between Claude 4.6 Sonnet rank-1 ICD-10 chapter and the 3-specialist gold standard . Interpreted per Landis \& Koch (1977): \<=0.20 slight; 0.21-0.40 fair; 0.41-0.60 moderate; 0.61-0.80 substantial; \>0.80 almost perfect . Range: -1.00 to 1.00
Time frame
At the time of algorithmic evaluation (June-July 2026)3
Top-3 Diagnostic Accuracy of GPT-4o
Proportion of cases in which the 3-specialist gold standard diagnosis appears within GPT-4o's ranked list of three differential diagnoses . Range: 0 to 1.00
Time frame
At the time of algorithmic evaluation (June-July 2026)4
Top-3 Diagnostic Accuracy of Claude 4.6 Sonnet
Proportion of cases in which the 3-specialist gold standard diagnosis appears within Claude 4.6 Sonnet's ranked list of three differential diagnoses\[cite: 1\]. Range: 0 to 1.00
Time frame
At the time of algorithmic evaluation (June-July 2026)Sponsors and contacts
Click on the lead sponsor to view all of their trials.
Source ClinicalTrials.gov