Computer-Aided Diagnostics

Designing computer-aided diagnostic systems has been a major focus of AI research since the advent of the field. The earliest attempts at this¹ used hand-curated knowledge bases. In these systems, expert doctors would be solicited to write down causal inference rules (see, for example, Figure 8-1).

There was basic support for uncertainty handling through certainty factors.

Figure 8-1. MYCIN was an early expert system used to diagnose bacterial infections. This is an example of a MYCIN rule for inference (adapted from the University of Surrey).

These rules were combined using a logical engine. A number of efficient inference techniques were designed that could effectively combine large databases of rules. Such systems were traditionally called “expert systems.”

Expert systems for medicine had a good run. Some of them were deployed widely and adopted internationally as well.² However, these systems failed to achieve significant traction with everyday doctors and nurses. One problem was that they were very finicky and hard to use. They also required their users to be able to pass in patient information in a highly structured format. Given that computers had barely penetrated standard clinics at the time, requiring highly specialized training for doctors and nurses proved to be too big an ask.

Probabilistic Diagnoses with Bayesian Networks

Another major problem with expert system tools was that they could only provide deterministic predictions. These deterministic predictions didn’t leave much room for uncertainty. What if the doctor was seeing a tricky patient where the diagnosis wasn’t clear? For a time, it seemed that if expert systems could be modified to account for uncertainties, this would allow them to achieve success.

This basic insight triggered a host of work on Bayesian networks for clinical diagnoses. (One of the authors of this book spent a year working on such a system as an undergrad.) However, these systems suffered from many of the same limitations as the expert systems. It was still necessary to solicit structural knowledge from doctors, and designers of Bayesian clinical networks faced the additional challenge of soliciting meaningful probabilities from doctors. This process added significant overhead to the process of adoption.

In addition, training a Bayesian network can be complicated. Different types of Bayesian network require different algorithms. Contrast this with deep learning algorithms, where gradient descent techniques work on almost all networks you can find. Robustness of learning is often what enables widespread adoption. This basic insight triggered a host of work on Bayesian networks for clinical diagnoses. (See Figure 8-2 for a simple example of a Bayesian network.)

Figure 8-2. A simple example of a Bayesian network for inferring whether the grass is wet at a given spot. (Source: Wikimedia.)

Electronic Health Record Data

Traditionally, hospitals maintained paper charts for their patients. These charts would record the tests, medications, and other treatments of the patient, allowing doctors to track the patient’s health with a quick glance at the chart. Unfortunately, paper health records had a host of difficulties associated with them. Transferring records between hospitals required a major amount of work, and it wasn’t easy to index or search paper health record data.

For this reason, there has been a major push over the last few decades in a number of countries to move from paper records to electronic health records (EHRs). In the US, the adoption of the Affordable Care Act significantly accelerated their adoption, and most major US health providers now store their patient records on EHR systems.

The broad adoption of EHR systems has spurred a boom in research on machine learning systems that work with EHR data. These systems aim to use large datasets of patient records to train models that will be capable of predicting things such as patient outcomes or risks. In many ways, these EHR models are the intellectual successors of the expert systems and Bayesian networks we just learned about. Like these earlier systems, EHR models seek to aid the process of diagnosis. However, while earlier systems sought to aid doctors in making real-time diagnoses, these newer systems content themselves (mostly) with working on the backend.

A number of projects have attempted to learn robust models from EHR data. While there have been some notable successes, learning on EHR data remains challenging for practitioners. Due to privacy concerns, there aren’t many large public EHR datasets available. As a result, only a small group of elite researchers have been able to design these systems thus far. In addition, EHR data tends to be very messy. Since human doctors and nurses manually enter information, most EHR data suffers from missing fields and all sorts of different conventions. Creating robust models that deal with the missing data has proven challenging.

However, this state of affairs is starting to change. Improved tools, both for preprocessing and for learning, have started to enable effective learning to occur on EHR systems. The DeepPatient system trains a denoising autoencoder on patient medical records to create a patient representation which it then uses to predict patient outcomes.⁵ In this system, a patient’s record is transformed from a set of unordered textual information into a vector. This strategy of transforming disparate data types into vectors has been widely successful throughout deep learning and seems poised to offer meaningful improvements in EHR systems as well. A number of models based on EHR systems have sprouted in the literature, many of which are starting to incorporate the latest tools of deep learning, such as recurrent networks or reinforcement learning. While models with these latest bells and whistles are still maturing, they’re very exciting and provide pointers to where the field is likely to head over the next few years.

Deep Radiology

Radiology is the science of using medical scans to diagnose disease. There are a variety of different scans that doctors use, such as MRI scans, ultrasounds, X-rays, and CT scans. For each of these, the challenge is to diagnose the state of the patient from the given scan imagery. This looks like a challenge well suited for convolutional learning methods. As we have seen in the previous chapters, deep learning methods are capable of learning sophisticated functions from image data. Much of modern radiology (the mechanical parts at least) consists of classifying and handling complex medical image data. The use of scans has a long and storied history in medicine (see Figure 8-4 for an example of an early X-ray).

In this section, we’ll quickly introduce a number of different types of scans and briefly cover some deep learning applications. Many of these applications are qualitatively similar. They start by obtaining a large enough dataset of scans from a medical institution. These scans are used to train a convolutional architecture (see Figure 8-3). Often, the architecture is a standard VGG or ResNet architecture, but sometimes with some tweaks to the core structure. The trained model often (at least according to perhaps naive statistics) has strong performance on the task in question.

Figure 8-3. This diagram draws out some standard convolutional architectures (VGG-19, Resnet-34, the number indicates the number of convolutions applied). These architectures are standard for image tasks and are commonly used for medical applications.

These advances have led to some perhaps inflated expectations. Some high-profile AI scientists—most notably, Geoff Hinton—have commented that deep learning for radiology will advance so far that it will no longer be worth training new radiologists in the near future.⁷ Is this actually right? There have been a string of recent advances in which deep learning systems have achieved what appears like near-human performance. However, these results come with many caveats, and these systems are often brittle in unknown fashions.

Our opinion is that the risk of direct 1-1 replacement of doctors remains low, but there is a real risk of systematic displacement. What does this mean? New startups are working to invent new business models in which deep learning systems do the large majority of scan analysis, with only a few doctors remaining.

X-Ray Scans and CT Scans

Informally, an X-ray scan—radiography, if we’re being precise—involves using X-rays to view some internal structure in the body (Figure 8-4). Computed tomography (CT) scans are a variant of X-ray scans in which the X-ray source and detectors rotate around the object being imaged, allowing for 3D images.

Figure 8-4. The first medical X-ray taken by Wilhelm Röntgen of his wife Anna Bertha Ludwig’s hand. The science of X-rays has come a long way since this first photograph, and there’s a chance that deep learning will take it much further yet!

A common misconception is that X-ray scans are only capable of imaging “hard” objects such as bones. This turns out to be quite false. CT scans are routinely used to image tissues in the body such as the brain (Figure 8-5), and backscatter X-rays are often used in airports to image travelers at security checkpoints. Mammograms use low-energy X-rays to scan breast tissue as well.

Figure 8-5. CT scan of a human brain from bottom to top. Note the capacity of CT scans to provide 3D information. (Source: Wikimedia.)

It’s worth noting that all X-ray scans are known to be linked to cancer, so a common goal is to minimize the exposure of patients to radiation by limiting the number of scans required. This risk is more marked for CT scans, which have to expose the patient for longer time periods in order to gather sufficient data. A wide variety of signal processing algorithms have been designed to reduce the number of scans required for CT. Some exciting recent work has started to use deep learning to further tune this reconstruction process so even fewer scans are required.

However, most uses of deep learning in the space are used to classify scans. For example, convolutional networks have been used to classify Alzheimer progression from CT brain images.⁸ Other work has claimed the ability to diagnose pneumonia from chest X-ray scans at near physician-level accuracy.⁹ Deep learning has similarly been used to achieve strong classification accuracy on mammography.¹⁰

Human-Level Accuracy Is Tricky!

When a paper claims that its system achieves near-human accuracy, it’s worth pausing to consider what that means. Usually, the authors of the paper choose some metric (say, ROC AUC), and a group of external physicians works to annotate the chosen test set for the study. The accuracy of the model on this test set is then compared against that of the “average” physician (often the mean or median of the physician scores).

This is a fairly complex process, and there are a number of ways in which this comparison can go wrong. First, the choice of metric can play a role—all too commonly, varying the choice of metric can lead to differences. Good analyses will consider multiple different metrics to ensure that the conclusions are robust to such variance.

Another point to note is that there’s considerable variation between doctors themselves. It’s worth checking to make sure that your choice of “average” is robust. A better metric might be to ask whether your algorithm is capable of beating the “best” doctor in the panel.

A third issue is that it can be extremely tricky to make sure that the test set isn’t “polluted.” (See our warning in Chapter 7.) Subtle forms of pollution can occur in which the scans from the same patient accidentally end up in both the training and test sets. If your model has very high accuracy, it’s worth double- and triple-checking for such leakages. All of us have been guilty of making mistakes on these pipelines in the past.

Finally, “human-level accuracy” doesn’t often mean much. As we’ve noted, some expert systems and Bayesian networks achieved human-level accuracy on limited tasks but failed to have a broad impact on medicine. The reason is that doctors perform a whole range of tasks which are tightly wound together. A given doctor may underperform a deep network on scan reading, but may be able to offer a much better diagnosis using other information. It’s worth remembering that these tasks are often synthetic, and may not match best physician practices. Prospective clinical trials using deep learning systems live with consenting patients will be needed to more accurately gauge the effectiveness of these techniques.

Learning Models as Therapeutics

So far in this chapter, we’ve seen that learning models can be effective assistants to doctors, helping aid the process of diagnosis and scan understanding. However, there’s some exciting evidence that learning models can move past being assistants to doctors to being therapeutic instruments in their own right.

How could this possibly work? One of the greatest powers of deep learning is that it is now feasible for the first time to build practical software that operates on perceptual data. For this reason, machine learning systems could potentially serve as “eyes” and “ears” to differently abled patients. A visual system could help patients with visual impairments more effectively navigate the world. An audio processing system could help patients with hearing impairments more effectively navigate the world. These systems face a number of challenges that other deep models don’t, since they have to operate effectively in real time. All the models we’ve considered so far in this book have been batch systems, suited for deployment on a backend server, not models fit for deployment on a live embedded device. There’s a whole host of challenges in dealing with machine learning in production which we won’t get into here, but we encourage interested readers to dive into the subject more deeply.

We also note that there’s a separate class of software-driven therapeutics that make uses of the powerful effects of modern software on the human brain. A groundswell of recent research has shown that modern software applications such as Facebook, Google, WeChat, and the like can be highly addictive. These apps are designed with bright colors and intended to hit many of the same centers in our brains as casino slot machines. There’s growing recognition that digital addictionis a real problem facing many patients.¹¹ This is a broad area beyond the scope of this book, but we note that there’s evidence that this power of modern software can be used for good too. Some software apps have been developed that use the psychological effects of modern apps as therapeutic interventions for patients struggling with depression or other conditions.

Diabetic Retinopathy

So far in this chapter, we have discussed applications of deep learning to medicine in a theoretical sense. In this section, we’ll roll up our sleeves and get our hands dirty with a practical example. In particular, we’re going to build a model to help diagnose diabetic retinopathy patient progression.

Diabetic retinopathy is a condition in which diabetes damages the health of the eyes. It is a major cause of blindness, especially in the developing world. The fundus is the interior area of the eye that’s opposite to the lens. A common strategy for diagnosis of diabetic retinopathy is for doctors to view an image of the patient’s fundus and label it manually. Significant work has gone into “fundus photography,” which develops techniques to capture patient fundus images (see Figure 8-6).

Figure 8-6. An image of a patient fundus from a patient who has undergone scatter laser surgery treatment for diabetic retinopathy. (Source: Wikimedia.)

The learning challenge for diabetic retinopathy is to design an algorithm that can classify a patient’s disease progress given an image of the patients’ fundus. At present, making such predictions requires skilled doctors or technicians. The hope is that a machine learning system could accurately predict disease progression from patient fundus images. This could provide patients with a cheap method of understanding their risk, which they could use before consulting a more expensive expert doctor for a diagnosis.

In addition, unlike EHR data, fundus images don’t contain much sensitive information about patients, which makes it easier to gather large fundus image datasets. For these reasons, a number of machine learning studies and challenges have been conducted on diabetic retinopathy datasets. In particular, Kaggle sponsored a contest aimed at creating good diabetic retinopathy models and put together a dataset of high-resolution fundus images. In the remainder of this section, you will learn how to use DeepChem to build a diabetic retinopathy classifier on the Kaggle Diabetic Retinopathy (DR) dataset.

The first step to working with this data is to preprocess and load the raw data. In particular, we crop each image to focus on its center square containing the retina. We then resize this center square to be of size 512 by 512.

The core data is stored in a set of directories on disk. We use DeepChem’s ImageLoader class to load these images from disk. If you’re interested, you can look through this loading and preprocessing code in detail, but we’ve wrapped it into a convenience helper function. In the style of the MoleculeNet loaders, this function also does a random training, validation, and test split:

train, valid, test = load_images_DR(split='random', seed=123)

Now that we have the data for this learning task, let’s build a convolutional architecture to learn from this dataset. The architecture for this task is fairly standard and resembles other architectures you’ve already seen in this book, so we don’t replicate it here. Here’s the invocation of the object wrapper for the underlying convolutional network:

# Define and build model
model = DRModel(
    n_init_kernel=32,
    batch_size=32,
    learning_rate=1e-5,
    augment=True,
    model_dir='./test_model')

This code sample defines a diabetic retinopathy convolutional network in DeepChem. As we will see later, training this model will take some heavy computation. For that reason, we recommend that you download our pretrained model from the DeepChem website and use that for your early exploration. We have already trained this model on the full Kaggle Diabetic Retinopathy dataset and stored its weights for your convenience. You can use the following commands to download and store the model (note that the first command should be entered on a single line, with no space after the +/+):

wget https://s3-us-west-1.amazonaws.com/deepchem.io/featurized_datasets
  /DR_model.tar.gz 
mv DR_model.tar.gz test_model/
cd test_model
tar -zxvf DR_model.tar.gz
cd ..

You can then restore the trained model weights as follows:

model.build()
model.restore(checkpoint="./test_model/model-84384")

We are restoring a particular pretrained “checkpoint” from this model. We provide more details on the restoration process and the full scripts used to achieve it in the code repository associated with this book. With the pretrained model in place, we can compute some basic statistics upon it:

metrics = [
    dc.metrics.Metric(DRAccuracy, mode='classification'),
    dc.metrics.Metric(QuadWeightedKappa, mode='classification')
]

There are a number of metrics that are useful for evaluating diabetic retinopathy models. Here we use, DRAccuracy, which is simply the model accuracy (percent of labels which are correct), and Cohen’s Kappa, a statistic used to measure agreement between two classifiers. This is useful because the diabetic retinopathy learning task is a multiclass learning problem.

Let’s evaluate our pretrained model on the test set with our metrics:

model.evaluate(test, metrics)

This produces the following results:

computed_metrics: [0.9339595787076572]
computed_metrics: [0.8494075470551462]

The basic model gets 93.4% accuracy on our test set. Not bad! (It’s important to note that this isn’t the same as the Kaggle test set—we’ve simply partitioned Kaggle’s training set into train/valid/test sets for our experimentation. You’re welcome to try submitting your trained model to Kaggle for evaluation on their test set, though.) Now, what if you’re interested in training the full model from scratch? This will take about a day or two’s training on a good GPU system, but is straightforward enough to do:

for i in range(10):
  model.fit(train, nb_epoch=10)
  model.evaluate(train, metrics)
  model.evaluate(valid, metrics)
  model.evaluate(valid, cm)
  model.evaluate(test, metrics)
  model.evaluate(test, cm)

We train the model for 100 epochs, pausing periodically to print out results from the model. If you’re running this job, we recommend making sure that your machine won’t shut down or go to sleep halfway through the job. There’s nothing as irritating as losing a large job to a sleep screen!

Conclusion

In many ways, the application of machine learning to medicine has the potential to have greater impact than many of the other applications we’ve seen so far. These other applications may have shifted what you do at work, but machine learning healthcare systems will soon change your personal healthcare experiences, along with the experiences of millions if not billions of others. For this reason, it’s worth pausing and thinking through some of the ethical repercussions.