An artificial intelligence model built by Harvard Medical School and Massachusetts Eye and Ear scientists was shown to be significantly more accurate than doctors at diagnosing pediatric ear infections in the first head-to-head evaluation of its kind, the research team working to develop the model for clinical use reported.
According to a new study published Aug. 16 in Otolaryngology–Head and Neck Surgery, the model, called OtoDX, was more than 95 percent accurate in diagnosing an ear infection in a set of 22 test images compared with 65 percent accuracy among a group of clinicians consisting of ENTs, pediatricians, and primary care doctors, who reviewed the same images.
When tested in a data set of more than 600 inner ear images, the AI model had a diagnostic accuracy of more than 80 percent, representing a significant leap over the average accuracy of clinicians reported in medical literature.
Science. Medicine. Humanity.
Harvard Medicine magazine delivered to your inbox
The model utilizes a type of AI called deep learning and was built from hundreds of photographs collected from pediatric patients prior to undergoing surgery at Mass Eye and Ear for recurrent ear infections or fluid in the ears. The results signify a major step toward the development of a diagnostic tool that can one day be deployed to clinics to assist doctors during patient evaluations, according to the authors. An AI-based diagnostic tool can give providers, like pediatricians and providers at urgent care clinics, an additional test to better inform their clinical decision-making.
“Ear infections are incredibly common in children yet frequently misdiagnosed, leading to delays in care or unnecessary antibiotic prescriptions,” said lead study author Matthew Crowson, HMS assistant professor of otolaryngology–head and neck surgery and an otolaryngologist and artificial intelligence researcher at Mass Eye and Ear. “This model won’t replace the judgment of clinicians but can serve to supplement their expertise and help them be more confident in their treatment decisions.”
Common condition, yet difficult to diagnose
Ear infections occur from a buildup of bacteria inside the middle ear. According to the National Institute on Deafness and Other Communication Disorders, at least five out of six children in the United States have had at least one ear infection before the age of three. When left untreated, ear infections can lead to hearing loss, developmental delays, complications like meningitis, and, in some developing nations, death. Conversely, overtreating children when they don’t have an ear infection can lead to antibiotic resistance and render the medications ineffective against future infections. This is an important, significant public health problem.
To ensure the best outcomes for children, clinicians must diagnose ear infections as accurately and early as possible. However, previous studies suggest that the conventional diagnostic accuracy of ear infections in children from a physical exam is routinely below 70 percent, even with innovations to technology and clinical practice guidelines. The difficulty of evaluating a child who is struggling or crying during an examination, coupled with the general inexperience many doctors and urgent care providers have in ear evaluations, may explain the lower-than-expected diagnostic rate, according to Crowson.
“Since clinicians would rather stay on the side of caution, it’s pretty easy to see why parents typically walk out of urgent care with a prescription for antibiotics,” he said.
In 2021, Crowson collaborated with Michael Cohen, HMS assistant professor of otolaryngology–head and neck surgery and director of the Multidisciplinary Pediatric Hearing Loss Clinic at Mass Eye and Ear, and Christopher Hartnick, HMS professor of otolaryngology–head and neck surgery and director of the Division of Pediatric Otolaryngology at Mass Eye and Ear, to develop a more accurate method of diagnosing ear infections using a machine-learning algorithm. An artificial neural network was trained with high-resolution photographs of tympanic membranes collected directly from patients during ear procedures where infection can be seen. These photos represent a gold standard set of data compared to AI-based tools that rely on images collected from search engines. In a proof-of-concept study published last year, the model was found to be 84 percent accurate in detecting normal versus abnormal middle ears.
Human versus machine
In the new study, the researchers compared the accuracy of a refined model directly against clinicians. More than 639 images of tympanic membranes from children ages 18 years or younger who were undergoing surgery for tube placement or draining fluid from the ears were used to train the model. The images were tagged as either normal, infected, or having liquid behind the eardrum, as opposed to the normal or abnormal classification from the team’s earlier model. With the added segment, the model achieved a mean diagnostic accuracy of 80.8 percent.
A survey was then created asking clinicians and trainees of various medical specialties to view 22 new images of tympanic membranes and diagnose the ear as one of the three tagged categories. Whereas the machine-learning model correctly categorized more than 95 percent of the sample images, the average diagnostic score among 39 clinicians who responded to the survey was 65 percent. Moreover, pediatricians and family medicine and general internists correctly categorized 60.1 percent and 59.1 percent of images, respectively.
Bringing artificial intelligence to the clinic
Ongoing studies are underway to validate and refine the AI model. To date, more than 1,000 intraoperative images of tympanic membranes have been amassed at Mass Eye and Ear.
In collaboration with Mass General Brigham Innovation, OtoDX is currently being employed in a prototype device paired with a smartphone app. The device acts as a mini otoscope that would fit over the phone’s camera and allow clinicians to take photos of the inside of a child’s ear, upload them directly to the app, and receive a diagnostic reading in seconds. With further validation, OtoDX may provide another tool for clinicians to glean information in real time during an exam.
As feedback for the pilot is processed, Mass General Brigham Innovation will support the OtoDX team in exploring opportunities to commercialize this impactful tool to assist even more clinicians and their patients.
Co-authors on the study included Krish Suresh of Mass Eye and Ear and David Bates of Brigham and Women’s Hospital.
This study was supported in part by the National Institutes of Health’s Biomedical Informatics and Data Science Training Program (grant T15LM007092-30).
The technology is the subject of a pending patent application.
