DeepMind has partnered with Moorfields Eye Hospital to develop a diagnostic technology to bring to market, which can help to detect the early signs of sight-threatening conditions, which, through early detection, can usually be treated and often reversed.
What technology is currently used?
As is already known, Artificial Intelligence is already being used within this field with eye specialists using Optimal Coherence Tomography (OCT scans) to aid them in their diagnoses on a day-to-day basis. However, these scans have proven to have their limitations.
These 3D scans provide a detailed image of the back of the eye. However, this requires highly trained, expert analysis to interpret and to make accurate diagnoses, which is in short supply and therefore, this is a time-consuming process. This can cause major delays in treatment which in some cases can be sight-threatening, if, for instance a bleed was to appear at the back of the eye and was untreated due to excess time waiting for OCT scan results.
What is this new AI technology?
According to the official report, the idea of this new technology is that it will analyse the ‘basic’ OCT scan extremely efficiently (taking just 30 seconds in one demonstration) and will identify from the images a wide range of eye diseases along with recommendations of how they should be treated and the urgency of each individual patient’s situation.
This is solving an issue long discussed within this sector; the ‘black box problem’. This is because for the first time, the AI is providing context to the professionals to aid in their final decision; it not only presents the findings but explains how it has arrived at this diagnosis and recommendation. This is in turn leaving room for the specialist to scrutinise the information which is provided to an appropriate level, so as to ensure the best treatment plan for each specific patient is reached. This is aided by the fact that the technology provides a confidence level of its findings in the form of a percentage which will determine the degree to which the specialist will need to question the result.
How does the technology work?
The technology uses two neural networks with an easily interpretable representation between them which has been split up into 5 distinct steps.
It starts with the raw OCT scan with which the segmentation network uses a 3D U-Net architecture to translate the scan into a detailed tissue map, which includes 15 classes of detail including anatomy and pathology having been trained with 877 clinical OCT scans.
Next the classification network analyses the tissue map and as a primary outcome provides one of 4 referral recommendations currently in use at Moorfields Eye Hospital:
- Urgent
- Semi-urgent
- Routine
- Observation only
For the training of this classification network, DeepMind collated 14,884 OCT scans of 7621 patients sent to the hospital with symptoms suggestive of macular pathology (the macular being the central region of the retina at the posterior pole of the eye). The final step as previously mentioned includes providing possible diagnoses with a percentage confidence level allowing appropriate scrutiny of the result to take place when necessary.
Have there been any challenges to overcome?
A major challenge in the use of OCT image segmentation (step 2 of 5 mentioned above) is the presence of ambiguous regions where the true tissue type cannot be deduced from the image meaning that there are multiple plausible interpretations which exist.
To overcome this, multiple instances of the segmentation network have been trained with each network instance creating a fully detailed segmentation map allowing multiple hypothesis to be given. As a result, doctors can analyse the evidence and select the most applicable hypothesis for each individual patient. Interestingly, the report states that in areas with clear image structures, these network instances may agree and provide similar results, and that videos can even be produced showing these results with further clarity.
Testing and quality control:
Achieving ‘expert performance on referral decisions’ is the main outcome which is hoped for with this project as an incorrect decision, in some cases, could lead to rapid sight-loss for a patient.
Within the report made about this project, DeepMind has stated that to optimise quality, first they needed to define what a Gold Standard of service is.
“This used information not available at the first patient visit and OCT scan, by examining the patient clinical records to determine the final diagnosis and optimal referral pathway in the light of that (subsequently obtained) information. Such a gold standard can only be obtained retrospectively”
Not including the people used in the training data set, Gold Standard labels were acquired for 997 patients and these were the patients which were used for testing the framework. Each patient received a referral suggestion from the network as well as independent referral suggestions from 8 specialists, 4 of whom were retina specialists and 4 of whom were optometrists trained in the retina.
Each of the 8 specialists provided 2 recommendations, one simply using the OCT scan and one using the OCT scan as well as the fundus imaging and clinical notes and these were given during separate sessions spaced at least two weeks apart.
The performance of the framework matched the referrals of 2 of the retinal specialists and exceeded those of the other 6 specialists when they were simply using the OCT imaging. When they had full access to notes and fundus images their performance improved but this time the framework matched the performance of 5 of the specialists but still continued to outperform the remaining 3. Overall, the framework performed very well not making any serious clinical errors.
As I have previously stated, the framework uses an ensemble of 5 segmentation model instances, but it also uses 5 classification model instances, and this is clinically proven to lead to more accurate results. In the testing process, it was found that when deriving a score for our framework and the experts as a weighted average of all wrong diagnoses, the framework achieved a lower average penalty point score than any of the experts.
Generalising this technology?
What is exciting about this two-stage framework is how the second network (classification network) is independent of the segmentation network. Therefore, to use it on another device, only the segmentation stage would need to be retrained so that the system knows how each tissue type appears on the scan image.
This has been tested on different devices which perform OCT scans, including Spectralis, from Heidelberg Engineering, which is the second most used device type at Moorfields Eye Hospital. It achieved an erroneous referral suggestion in only 4 of the 116 cases tested, similar to the 5.5% error rate of device 1.
A first step however in getting this technology to a more widespread audience, is its application in all Moorfields Hospitals in the UK for the first 5 years if and when it is launched, free of charge! As well as this, due to the fact it can be applied to a wider range of devices, it will be much cheaper for other organisations to apply the technology to their existing devices rather than having to purchase additional specialist equipment.
In all, this could be a giant leap for many countries around the world, hugely increasing the number of people who could potentially benefit from this innovative technology rather than limiting itself to one device and therefore limiting who can use it.
Overall, according to DeepMind, the initial 14 months of research and development have been very successful, and this technology is by no means ready to be launched into the marketplace. However, with more rigorous testing and technological developments, before long it is certain that we will be seeing this technology implemented nationally and possibly even internationally. We hope that with this framework as a basis, in the future, others will be able to build on this work, branching out into new areas of this sector now that the ‘Black Box’ problem has been identified and somewhat overcome. Additionally, looking outside the box, many have suggested that this technology could even be useful for training purposes as people would be able to use the technology to learn how to interpret images of the eye and how and where to look for aspects within the image creating cause for concern.
It is said that this could be a revolutionary step forward in this field and that much like 5G will allow people in rural áreas around the world to gain Access to healthcare specialists, this AI technology will provide the fastes and most accurate diagnoses no matter where someone is situated in the world.
Don’t miss out on a single post. Subscribe to LUCA Data Speaks.
