1. INTRODUCTION

Over the past few decades, medical imaging techniques, such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), mammography, ultrasound, and X-ray, have been used for the early detection, diagnosis, and treatment of diseases (1). In the clinic, medical image interpretation has been performed mostly by human experts such as radiologists and physicians. However, given wide variations in pathology and the potential fatigue of human experts, researchers and doctors have begun to benefit from computer-assisted interventions. Although the rate of progress in computational medical image analysis has not been as rapid as that in medical imaging technologies, the situation is improving with the introduction of machine learning techniques.

In applying machine learning, finding or learning informative features that well describe the regularities or patterns inherent in data plays a pivotal role in various tasks in medical image analysis. Conventionally, meaningful or task-related features were designed mostly by human experts on the basis of their knowledge about the target domains, making it challenging for nonexperts to exploit machine learning techniques for their own studies. In the meantime, there have been efforts to learn sparse representations based on predefined dictionaries, possibly learned from training samples. Sparse representation is motivated by the principle of parsimony in many areas of science; that is, the simplest explanation of a given observation should be preferred over more complicated ones. Sparsity-inducing penalization and dictionary learning have demonstrated the validity of this approach for feature representation and feature selection in medical image analysis (2–6). It should be noted that sparse representation or dictionary learning methods described in the literature still find informative patterns or regularities inherent in data with a shallow architecture, thus limiting their representational power. However, deep learning (7) has overcome this obstacle by incorporating the feature engineering step into a learning step. That is, instead of extracting features manually, deep learning requires only a set of data with minor preprocessing, if necessary, and then discovers the informative representations in a self-taught manner (8, 9). Therefore, the burden of feature engineering has shifted from humans to computers, allowing nonexperts in machine learning to effectively use deep learning for their own research and/or applications, especially in medical image analysis.

The unprecedented success of deep learning is due mostly to the following factors: (a) advances in high-tech central processing units (CPUs) and graphics processing units (GPUs), (b) the availability of a huge amount of data (i.e., big data), and (c) developments in learning algorithms (10–14). Technically, deep learning can be regarded as an improvement over conventional artificial neural networks (15) in that it enables the construction of networks with multiple (more than two) layers. Deep neural networks can discover hierarchical feature representations such that higher-level features can be derived from lower-level features (9). Because these techniques enable hierarchical feature representations to be learned solely from data, deep learning has achieved record-breaking performance in a variety of artificial intelligence applications (16–23) and grand challenges (24, 25; see https://grand-challenge.org). In particular, improvements in computer vision prompted the use of deep learning in medical image analysis, such as image segmentation (26, 27), image registration (28), image fusion (29), image annotation (30), computer-aided diagnosis (CADx) and prognosis (31–33), lesion/landmark detection (34–36), and microscopic image analysis (37, 38).

Deep learning methods are highly effective when the number of available samples during the training stage is large. For example, in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), more than one million annotated images were available (24). However, in most medical applications there are far fewer images (i.e., <1,000). Therefore, a primary challenge in applying deep learning to medical images is the limited number of training samples available to build deep models without suffering from overfitting. To overcome this challenge, research groups have devised various strategies, such as (a) taking either two-dimensional (2D) or three-dimensional (3D) image patches, rather than the full-sized images, as input (29, 39–45) in order to reduce input dimensionality and thus the number of model parameters; (b) expanding the data set by artificially generating samples via affine transformation (i.e., data augmentation), and then training their network from scratch with the augmented data set (39–42); (c) using deep models trained on a huge number of natural images in computer vision as “off-the-shelf” feature extractors, and then training the final classifier or output layer with the target-task samples (43, 45); (d) initializing model parameters with those of pretrained models from nonmedical or natural images, then fine-tuning the network parameters with the task-related samples (46, 47); and (e) using models trained with small-sized inputs for arbitrarily sized inputs by transforming weights in the fully connected layers into convolutional kernels (36, 48).

In terms of input types, we can categorize deep models as typical multilayer neural networks that take vector-format (i.e., nonstructured) values as input and convolutional networks that take 2D or 3D (i.e., structured) values as input. Because of the structural characteristics of images (the structural or configural information contained in neighboring pixels or voxels is another important source of information), convolutional neural networks (CNNs) have attracted great interest in the field of medical image analysis (26, 35–37, 48–50). However, networks with vectorized inputs have also been successfully used in different medical applications (28, 29, 31, 33, 51–54). Along with deep neural networks, deep generative models (55)—such as deep belief networks (DBNs) and deep Boltzmann machines (DBMs), which are probabilistic graphical models with multiple layers of hidden variables—have been successfully applied to brain disease diagnosis (29, 33, 47, 56), lesion segmentation (36, 49, 57, 58), cell segmentation (37, 38, 59, 60), image parsing (61–63), and tissue classification (26, 35, 48, 50).

This review is organized as follows. In Section 2, we explain the computational theories of neural networks and deep models [e.g., stacked auto-encoders (SAEs), DBNs, DBMs, CNNs] and discuss how they extract high-level representations from data. In Section 3, we introduce recent studies using deep models for different applications in medical imaging, including image registration, anatomy localization, lesion segmentation, detection of objects and cells, tissue segmentation, and computer-aided detection (CADe) and CADx. Finally, in Section 4 we conclude by summarizing research trends and suggesting directions for further improvements.

2. DEEP LEARNING

In this section, we explain the fundamental concepts of feed-forward neural networks and basic deep models in the literature. We focus on learning hierarchical feature representations from data. We also discuss how to efficiently learn parameters of deep architecture by reducing overfitting.

2.1. Feed-Forward Neural Networks

In machine learning, artificial neural networks are a family of models that mimic the structural elegance of the neural system and learn patterns inherent in observations. The perceptron (64) is the earliest trainable neural network with a single-layer architecture,¹ composed of an input layer and an output layer. A perceptron, or a modified perceptron with multiple output units (Figure 1a), is regarded as a linear model, prohibiting its application in tasks involving complicated data patterns, despite the use of nonlinear activation functions in the output layer.

This limitation can be overcome by introducing a so-called hidden layer between the input layer and the output layer. Note that in neural networks the units of the neighboring layers are fully connected to one another, but there are no connections among units in the same layer. For a two-layer neural network (Figure 1b), also known as a multilayer perceptron, given an input vector , we can write the estimation function of an output unit y_k as a composition function as follows:

where the superscript denotes a layer index, f⁽¹⁾(·) and f⁽²⁾(·) denote nonlinear activation functions of units at the specified layers, M is the number of hidden units, and Θ={W⁽¹⁾, W⁽²⁾, b⁽¹⁾, b⁽²⁾} is a parameter set.² Conventionally, the hidden units’ activation function, f⁽¹⁾(·), is commonly defined with a sigmoidal function such as a logistic sigmoid function or a hyperbolic tangent function, whereas the output units’ activation function f⁽²⁾(·) is dependent on the target task. Because the estimation proceeds in a forward direction, this type of network is also referred to as a feed-forward neural network.

When the hidden layer in Equation 1 is regarded as a feature extractor, from an input v, the output layer is only a simple linear model,

where ϕ_j(v)≡f⁽¹⁾(∑^D_i=1W⁽¹⁾_jiv_i+b⁽¹⁾_j). The same interpretation holds when there is a higher number of hidden layers. Thus, it is intuitive that the role of hidden layers is to find features that are informative for the target task.

The practical use of neural networks requires that the model parameters Θ be learned from data. The problem of parameter learning can be formulated as the minimization of the error function. From an optimization perspective, the error function E for neural networks is highly nonlinear and nonconvex. Thus, there is no analytic solution of the parameter set Θ. Instead, one can use a gradient descent algorithm by updating the parameters iteratively. In order to utilize a gradient descent algorithm, there must be a way to compute a gradient ∇E(Θ) evaluated at the parameter set Θ.

For a feed-forward neural network, the gradient can be efficiently evaluated by means of error back-propagation (65). Once the gradient vector of all the layers is known, the parameters Θ ∈{W⁽¹⁾, W⁽²⁾, b⁽¹⁾, b⁽²⁾} can be updated as follows:

where η is a learning rate and τ denotes an iteration index. The update process is repeated until convergence or until the predefined number of iterations is reached. As for the parameter update in Equation 3, the stochastic gradient descent with a small subset of training samples, termed a minibatch, is commonly used in the literature (66).

2.2. Deep Models

Under a mild assumption on the activation function, a two-layer neural network with a finite number of hidden units can approximate any continuous function (67); therefore, it is regarded as a universal approximator. However, it is also possible to approximate complex functions to the same accuracy by using a deep architecture (i.e., one with more than two layers), with a far fewer number of units (8). Thus, it is possible to reduce the number of trainable parameters, enabling training with a relatively small data set (68).

2.3. Unsupervised Feature Representation Learning

Compared with shallow architectures that require a good feature extractor designed mostly by hand on the basis of expert knowledge, deep models are useful for discovering informative features from data in a hierarchical manner (i.e., from fine to abstract). Here, we introduce three deep models that are widely used in different applications for unsupervised feature representation learning.

An auto-encoder or auto-associator (69) is a special type of two-layer neural network that learns a latent or compressed representation of the input by minimizing the reconstruction error between the input and output values of the network, namely the reconstruction of the input from the learned representations. Because of its simple, shallow structure, a single-layer auto-encoder's representational power is very limited. But when multiple auto-encoders are stacked (Figure 2a) in a configuration called an SAE, one can significantly improve the representational power by using the activation values of the hidden units of one auto-encoder as the input to the next higher auto-encoder (70). One of the most important characteristics of SAEs is their ability to learn or discover highly nonlinear and complicated patterns, such as the relations among input values. When an input vector is presented to an SAE, the different layers of the network represent different levels of information. That is, the lower the layer in the network is, the simpler the patterns are, and the higher the layer is, the more complicated or abstract the patterns inherent in the input vector are.

With regard to training parameters of the weight matrices and the biases in SAE, a straightforward approach is to apply back-propagation to the gradient-based optimization technique, beginning from random initialization by using the SAE as a conventional feed-forward neural network. Unfortunately, deep networks trained in this manner perform worse than networks with a shallow architecture, as they fall into a poor local optimum (71). To circumvent this problem, one should consider greedy layer-wise learning (10, 72). The key idea of greedy layer-wise learning is to pretrain one layer at a time. That is, the user trains parameters of the first hidden layer with the training data as input, and then trains parameters of the second hidden layer with the output from the first hidden layer as input, and so on. In other words, the representation of the lth hidden layer is used as input for the (l+1)-th hidden layer. An important advantage of such a pretraining technique is that it is conducted in an unsupervised manner with a standard back-propagation algorithm, enabling the user to increase the size of the data set by exploiting unlabeled samples for training.

A restricted Boltzmann machine (RBM) (73) is a single-layer undirected graphical model with a visible layer and a hidden layer. It assumes symmetric connectivities between visible and hidden layers, but no connections among units within the same layer. Because of the symmetry of the connectivities, it can generate input observations from hidden representations. Therefore, an RBM naturally becomes an auto-encoder (10, 73), and its parameters are usually trained by use of a contrastive divergence algorithm (74) so as to maximize the log likelihood of observations. Like SAEs, RBMs can be stacked in order to construct a deep architecture, resulting in a single probabilistic model called a DBN. A DBN has one visible layer v and a series of hidden layers h⁽¹⁾, …, h^(L) (Figure 2b). Note that when multiple RBMs are stacked hierarchically, although the top two layers still form an undirected generative model (i.e., an RBM), the lower layers form directed generative models. Thus, the joint distribution of the observed units v and the l hidden layers h^(l) (l=1, …, L) in DBN is

where P(h^(l)|h^(l+1)) corresponds to a conditional distribution for the units of layer l given the units of layer l+1, and P(h^(L−1), h^(L)) denotes the joint distribution of the units in layers L−1 and l.

Regarding the learning of parameters, the greedy layer-wise pretraining scheme (10) can be applied in the following steps.

1.	Train the first layer as an RBM with v=h(0).
2.	Use the first hidden layer to obtain the representation of inputs with either the mean activations of P(h(1)=1|h(0)) or samples drawn according to P(h(1)|h(0)), which will be used as observations for the second hidden layer.
3.	Train the second hidden layer as an RBM, taking the transformed data (mean activations or samples) as training examples (for the visible layer of the RBM).
4.	Iterate steps 2 and 3 for the desired number of layers, each time propagating upward either mean activations P(h(l+1)=1|h(l)) or samples drawn according to the conditional probability P(h(l+1)|h(l)).

After the greedy layer-wise training procedure is complete, one can apply the wake–sleep algorithm (75) to further increase the log likelihood of the observations. Usually, however, no further procedure is conducted to train the whole DBN jointly in practice.

A DBM (55) is also constructed by stacking multiple RBMs in a hierarchical manner. However, in contrast to DBNs, all the layers in DBMs form an undirected generative model following the stacking of RBMs (Figure 2c). Thus, for hidden layer l, except in the case of l=1 and l=L, the layer's probability distribution is conditioned by its two neighboring layers, l+1 and L−1 [i.e., P(h^(l)|h^(l+1), h^(l−1))]. The incorporation of information from both the upper and lower layers improves a DBM's representational power so that it is more robust to noisy observations.

Let us consider a three-layer DBM, namely the L=2 DBM shown in Figure 2c. Given the values of the units in the neighboring layer(s), the probability of either the binary visible or binary hidden units being set to one is computed as follows:

where σ(·) denotes a logistic sigmoid function. In order to learn the parameters Θ={W⁽¹⁾, W⁽²⁾},³ we maximize the log likelihood of the observations. The derivative of the log likelihood of the observations with respect to the model parameters takes the following simple form:

where denotes the data-dependent statistics obtained by sampling the model conditioned on the visible units v(=h⁽⁰⁾) and denotes the data-independent statistics obtained by sampling from the model. When the model approximates the data distribution well, data-dependent and data-independent statistics reach equilibrium.

2.4. Fine-Tuning Deep Models for Target Tasks

Note that, during feature representation learning for the three deep models described above, the target values (either discrete labels or continuous real values of observations) are never involved. Therefore, there is no guarantee that the representations learned by SAEs, DBNs, or DBMs are discriminative for a classification task, for example. To address this problem, the so-called fine-tuning step is generally followed after unsupervised feature representation learning.

For a certain task involving either classification or regression, it is straightforward to convert feature representation learning models into a deep neural network by stacking another output layer on top of the highest hidden layer in an SAE, DBN, or DBM with an appropriate output function. In the case of a DBM, the original input vector should first be augmented with the marginals of the approximate posterior of the second hidden layer as a by-product when converting a DBM into a deep neural network (55). The top output layer is then used to predict the target value(s) of an input. To fine-tune the parameters in a deep neural network, we first take the pretrained connection weights of the hidden layers as the initial values, randomly initialize the connection weights between the top hidden layer and the output layer, and then train the parameters jointly in a supervised (i.e., end-to-end) manner by gradient descent with a back-propagation algorithm. Initialization of the parameters via pretraining helps the supervised optimization reduce the risk of falling into poor local optima (10, 71).

2.5. Convolutional Neural Networks

In the deep models of SAEs, DBNs, and DBMs, described above, the inputs are always in vector form. However, for (medical) images, the structural information among neighboring pixels or voxels is also important, but vectorization inevitably destroys such structural and configural information in images. CNNs (76) are designed to better utilize spatial and configural information by taking 2D or 3D images as input. Structurally, CNNs have convolutional layers interspersed with pooling layers, followed by fully connected layers as in a standard multilayer neural network. Unlike a deep neural network, a CNN exploits three mechanisms—a local receptive field, weight sharing, and subsampling (Figure 3)—that greatly reduce the degrees of freedom in a model.

The role of a convolutional layer is to detect local features at different positions in the input feature maps with learnable kernels k^(l)_ij, namely connection weights between the feature map i at layer L−1 and the feature map j at layer l. Specifically, the units of the convolutional layer l compute their activation A^(l)_j on the basis of only a spatially contiguous subset of units in the feature maps A^(l−1)_i of the preceding layer L−1 by convolving the kernels k^(l)_ij as follows:

where M^(l−1) denotes the number of feature maps in layer L−1, the asterisk denotes a convolutional operator, b^(l)_j is a bias parameter, and f(·) is a nonlinear activation function. Due to the mechanisms of weight sharing and local receptive field, when the input feature map is slightly shifted, the activation of the units in the feature maps is shifted by the same amount.

A pooling layer follows a convolutional layer to downsample the feature maps of the preceding convolutional layer. Specifically, each feature map in a pooling layer is linked to a feature map in the convolutional layer; each unit in a feature map of the pooling layer is computed on the basis of a subset of units within a local receptive field from the corresponding convolutional feature map. Similar to the convolutional layer, the receptive field finds a representative value (e.g., maximum or average) among the units in its field. Usually, a change in the size of the receptive field during convolution is set equal to the size of the receptive field for subsampling, helping the CNN to be translation invariant.

Theoretically, the gradient descent method combined with a back-propagation algorithm can also be applied to the learning parameters of a CNN. However, due to the special mechanisms of weight sharing, the local receptive field, and pooling, slight changes need to be made; that is, one needs to sum the gradients for a given weight over all the connections using the kernel weights in order to determine which patch in the layer's feature map corresponds to a unit in the next layer's feature map, and to upsample the feature maps of the pooling layer to recover the reduced size of the maps.

2.6. Reducing Overfitting

A critical challenge in training deep models arises from the limited number of training samples compared with the number of learnable parameters. Thus, reducing overfitting has long presented a challenge. Recent studies have devised algorithmic techniques to better train deep models. Some of the techniques are as follows.

1.	Initialization/momentum (77, 78) involves the use of well-designed random initialization and a particular schedule of slowly increasing the momentum parameter as iteration passes.
2.	Rectified linear unit (ReLU) (12, 79, 80) applies to nonlinear activation functions.
3.	Denoising (11) involves stacking layers of denoising auto-encoders, which are trained locally to reconstruct the original “clean” inputs from the corrupted versions of them.
4.	Dropout (13) and DropConnect (81) randomly deactivate a fraction (e.g., 50%) of the units or connections in a network on each training iteration.
5.	Batch normalization (14) performs normalization for each minibatch and back-propagates the gradients through the normalization parameters.

See the references cited for further details.

3. APPLICATIONS IN MEDICAL IMAGING

Compared with other machine learning techniques in the literature, deep learning has witnessed significant advances. These successes have prompted researchers in the field of computational medical imaging to investigate the potential of deep learning in medical images acquired with, for example, CT, MRI, PET, and X-ray. In this section, we discuss the practical applications of deep learning in image registration and localization, detection of anatomical and cellular structures, tissue segmentation, and computer-aided disease prognosis and diagnosis.

3.1. Deep Feature Representation Learning in Medical Images

Many existing medical image processing methods rely on morphological feature representations to identify local anatomical characteristics. However, such feature representations were designed mostly by human experts, and the image features are often problem specific and not guaranteed to work for other image types. For instance, image segmentation and registration methods designed for 1.5-T T1-weighted brain MR images are not applicable to 7.0-T T1-weighted MR images (28, 52), not to mention to other modalities or different organs. Furthermore, 7.0-T MR images can reveal the brain's anatomy with a resolution equivalent to that obtained from thin slices in vitro (82). Thus, researchers can clearly observe fine brain structures at the micrometer scale, which previously was possible only with in vitro imaging. However, the lack of efficient computational tools substantially hinders the translation of new imaging techniques into the medical imaging arena.

Although state-of-the-art methods use supervised learning to find the most relevant and essential features for target tasks, they require a significant number of manually labeled training data, and the learned features may be superficial and may misrepresent the complexity of the anatomical structures. More critically, the learning procedure is often confined to a particular template domain, with a certain number of predesigned features. Therefore, once the template or image features change, the entire training process has to start over again. To address these limitations, Wu et al. (28, 52) developed a general feature representation framework that can (a) capture the intrinsic characteristics of anatomical structures necessary for accurate brain region segmentation and correspondence detection and (b) be flexibly applied to different kinds of medical images. Specifically, these authors used an SAE with a sparsity constraint, which they therefore termed a sparse auto-encoder, to hierarchically learn feature representations in a layer-by-layer manner. Their SAE model consisted of encoding and decoding modules hierarchically (Figure 4). In the encoding module, given an input image patch x, the model mapped the input to an activation vector y⁽¹⁾ through nonlinear deterministic mapping. The authors then repeated this procedure by using y⁽¹⁾ as the input to train the second layer, and so forth, until they obtained high-level feature representations (Figure 4). The decoding module was used to validate the expressive power of the learned feature representations by minimizing the reconstruction errors between the input image patch x and the reconstructed patch z after decoding.

Figure 5 demonstrates the power of feature representations learned by deep learning methods. Figure 5a–c shows a typical image registration result for brain images of an elderly patient, and Figure 5d–f compares different feature representations for finding a correspondence of a template point. Clearly, the deformed subject image in Figure 5c is far from being well registered with the template image in Figure 5a, especially for ventricles. It is very difficult to learn meaningful features given such inaccurate correspondences derived from imperfect image registration, a problem that many supervised learning methods suffer from (83–85). Moreover, the features [e.g., local patches and scale-invariant feature transform (SIFT) (86)] either detect too many noncorresponding points when using the entire intensity patch as the feature vector (Figure 5d) or have too-low responses and thus miss the correspondence when using SIFT (Figure 5e). Meanwhile, SAE-learned feature representations present the least confusing correspondence information for the subject point under consideration, making it easy to locate the correspondence of the template point in the subject image domain.

In order to qualitatively evaluate the registration accuracy, Wu et al. obtained deformable image registration results over various public data sets (Figure 6). Compared with the state-of-the-art registration methods of intensity-based diffeomorphic Demons (87) and feature-based HAMMER (88) for 1.5- and 3.0-T MR images, the SAE-learned feature representation depicted in Figure 6e performs better.

Another successful medical application involves localizing a prostate from MR images (89, 90). Accurate prostate localization in MR images is difficult for two reasons: (a) The appearance patterns around the prostate boundary vary significantly between patients, and (b) the intensity distributions vary between patients and often do not follow a Gaussian distribution. To address these challenges, Guo et al. (90) used an SAE to learn hierarchical feature representations from MR prostate images. The resulting learned features were integrated into a sparse patch-matching framework to find the corresponding patches in the atlas images for label propagation (91). Finally, a deformable model was employed to segment the prostate by combining the shape prior with the prostate likelihood map derived from sparse patch matching. Figure 7 shows typical prostate segmentation results from different patients, produced by three different feature representations.

The applications described above demonstrate that (a) the latent feature representations inferred by deep learning can successfully describe local image characteristics; (b) researchers can rapidly develop image analysis methods for new medical imaging modalities by using a deep learning framework to learn the intrinsic feature representations; and (c) the entire learning-based framework can be adapted to learn imaging feature representations and extended to various medical imaging applications, such as hippocampus segmentation (92) and prostate localization in MR images (89, 90).

3.2. Deep Learning for Detection of Anatomical Structures

Localization and interpolation of anatomical structures in medical images are key steps in the radiological workflow. Radiologists usually accomplish these tasks by identifying certain anatomical signatures, namely image features that can distinguish one anatomical structure from others. Is it possible for a computer to automatically learn such anatomical signatures? The success of such methods essentially depends on how many anatomy signatures can be extracted by computational operations. Whereas early studies often created specific image filters to extract anatomical signatures, more recent research has revealed that deep learning–based approaches have become prevalent for two reasons: (a) Deep learning technologies are now mature enough to solve real-world problems, and (b) more and more medical image data sets have become available to facilitate the exploration of big medical image data.

Shin et al. (51) used SAEs to separately learn both visual and temporal features in order to detect multiple organs in a time series of 3D dynamic contrast–enhanced MRI scans over data sets from two studies of liver metastases and one study of kidney metastases. Unlike conventional SAEs, the SAE in this study involved the application of a pooling operation after each layer so that features of progressively larger input regions were essentially compressed. Because different organ classes have different properties, the authors trained multiple models to separate each organ from all of the other organs in a supervised manner.

Roth et al. (93) presented a method for organ- or body part–specific anatomical classification of medical images using deep convolutional networks. Specifically, they trained their deep network by using 4,298 axial 2D CT images to learn five parts of the body: neck, lungs, liver, pelvis, and legs. Their experiments achieved an anatomy-specific classification error of 5.9% and an average AUC (area under the receiver-operating characteristic curve) value of 0.998. However, real-world applications may require more finely grained differentiation than that used for only five body parts (e.g., they may need to distinguish aortic arch from cardiac sections). To address this limitation, Yan et al. (94, 95) designed a multistate deep learning framework with a CNN to identify the body part of a transversal slice. Because each slice may contain multiple organs (enclosed in bounding boxes), the CNN was trained in a multi-instance fashion (96) in which the objective function was adapted such that, as long as one organ was correctly labeled, the corresponding slice was considered correct. Therefore, the pretrained CNN was sensitive to the discriminative bounding boxes. On the basis of the pretrained CNN's responses, discriminative and noninformative bounding boxes were selected to further boost the representation power of the pretrained CNN. At run time, a sliding-window approach was employed to apply the boosted CNN to the subject image. Because the CNN had peaky responses only on discriminative bounding boxes, it essentially identified body parts by focusing on the most distinctive local information. Compared with global image context-based approaches, this local approach was more accurate and robust. These authors’ body part recognition method was tested on 12 body parts on 7,489 CT slices, collected from scans of 675 patients varying in age from 1 to 90 years. The entire data set was divided into three groups: 2,413 (225 patients) for training, 656 (56 patients) for validation, and 4,043 (394 patients) for testing.

Digitized tissue histopathology has recently been employed for microscopic examination and automatic disease grading. A primary challenge in microscopic image analysis involves the need to analyze all individual cells for accurate diagnosis, given that the differentiation of most disease grades depends strongly on cell-level information. To address this challenge, researchers have employed deep CNNs to robustly and accurately detect and segment cells from histopathological images (37, 38, 53, 54, 97–99), which can significantly benefit cell-level analysis for cancer diagnosis.

In a pioneering study, Cireşan et al. (37) used a deep CNN to detect mitosis in breast cancer histology images. Their networks were trained to classify each pixel in the images from a patch centered on the pixel. Their method won the 2012 International Conference on Pattern Recognition (ICPR) Mitosis Detection Contest,⁴ outperforming other contestants by a significant margin.

Since then, different groups have used different deep learning methods for detection in histology images. For example, Xu et al. (54) used an SAE to detect cells on breast cancer histological images. To train their deep model, they utilized a denoising auto-encoder to improve robustness to outliers and noises. Su et al. (53) also used an SAE as well as sparse representation to detect and segment cells from microscopic images. Sirinukunwattana et al. (100) proposed a spatially constrained CNN (SC-CNN) to detect and classify nuclei in histopathology images. Specifically, they used an SC-CNN to estimate the likelihood of a pixel being the center of a nucleus, where pixels with high probability values were spatially constrained to locate in the vicinity of the center of nuclei. They also developed a neighboring ensemble predictor coupled with a CNN to more accurately predict the class label of the detected cell nuclei. Chen et al. (38) designed a deep cascaded CNN by exploiting the technique of the full CNN, which replaces the fully connected layers with all-convolutional kernels (101). They first trained a coarse retrieval model to identify and locate mitosis candidates while maintaining high sensitivity. On the basis of the retrieved candidates, they then created a fine discrimination model by transferring deep and rich feature hierarchies learned on a large natural image data set to distinguish mitoses from hard mimics. Their cascaded CNN achieved the best detection accuracy in the 2014 ICPR MITOS-ATYPIA challenge.⁵

3.3. Deep Learning for Segmentation

Automatic segmentation of brain images is a prerequisite for quantitative assessment of the brain in patients of all ages. An important step in brain image preprocessing involves removing nonbrain regions such as the skull. Although current methods demonstrate good results on nonenhanced T1-weighted images, they still struggle when applied to other modalities and pathologically altered tissues. To circumvent such limitations, Kleesiek et al. (27) used 3D convolutional deep learning architecture for skull extraction, a technique that was not limited to nonenhanced T1-weighted MR images. While training their 3D CNN, they constructed minibatches of multiple cubes that were larger than the actual input to their 3D CNN for computational efficiency. Specifically, their deep model could take an arbitrary-sized 3D patch as input by building a fully convolutional network (101); thus, the output could be a block of predictions per input, rather than a single prediction as in a conventional CNN. Over four different data sets, their method achieved the highest average specificity measures in comparison to six commonly used tools (i.e., BET, BEaST, BSE, ROBEX, HWA, and 3dSkullStrip), whereas its sensitivity was about average.

Moeskops et al. (102) devised a multiscale CNN to enhance robustness in neonatal image segmentation and spatial consistency. Their network used multiple patch sizes and multiple convolution kernel sizes to acquire multiscale information about each voxel. Using this method, the authors obtained promising segmentation results for eight tissue types, with a Dice ratio⁶ averaging 0.82 to 0.91 over five different data sets.

The first year of life is the most dynamic phase of postnatal human brain development, characterized by rapid tissue growth and development of a wide range of cognitive and motor functions. Accurate tissue segmentation of infant brain MR images into white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF) in this phase is crucial in studies of normal and abnormal early brain development. Segmentation of infants’ brain MR images is considerably more difficult to perform than for adults because of the reduced tissue contrast (103), increased noise, severe partial volume effect (104), and ongoing WM myelination (103, 105). Specifically, the WM and GM exhibit almost the same intensity level (especially in the cortical regions), resulting in low image contrast. Although many methods have been proposed for infant brain image segmentation, most focus on segmentation of images of either neonates (∼3 months) or infants (>12 months) using a single T1-weighted or T2-weighted image (106–110). Few studies have addressed the challenges posed by segmentation of isointense-phase images (around 6 months old).

To overcome these difficulties, Zhang et al. (26) designed four CNN architectures to segment infant brain tissues on the basis of multimodal MR images. Specifically, each CNN contained three input feature maps corresponding to T1-weighted, T2-weighted, and fractional anisotropy (FA) image patches measuring 13×13 voxels. The authors applied to each CNN three convolutional layers and one fully connected layer, followed by an output layer with a softmax function for tissue classification. On a set of manually segmented isointense-phase brain images, these CNNs significantly outperformed competing methods.

More recently, Nie et al. (48) proposed the use of multiple fully convolutional networks (mFCNs) (Figure 8) to segment isointense-phase brain images with T1-weighted, T2-weighted, and FA modality information. Instead of simply combining three-modality data from the original (low-level) feature maps, they employed a deep architecture to effectively fuse high-level information from all three modalities. They assumed that high-level representations from different modalities were complementary to one another. First, the authors trained one network for each modality in order to effectively employ information from multiple modalities; second, they fused multiple-modality features from the high layer of each network (Figure 8). In these experiments, the mFCNs achieved average Dice ratios of 0.852 for CSF, 0.873 for GM, and 0.887 for WM from eight subjects, outperforming fully convolutional networks and other competing methods.

3.4. Deep Learning for Computer-Aided Detection

The goal of CADe is to find or localize abnormal or suspicious regions in structural images, and thus to alert clinicians. CADe aims to increase the detection rate of diseased regions while reducing the false-negative rate, which may be due to error or fatigue on the part of the observers. Although CADe is well established in medical imaging, deep learning methods have improved its performance in different clinical applications.

Typically, CADe occurs as follows: (a) The candidate regions are detected by means of image processing techniques; (b) the candidate regions are represented by a set of features, such as morphological or statistical information; and (c) the features are fed into a classifier, such as a support vector machine (SVM), to output a probability or make a decision as to whether disease is present. As explained in Section 1, human-designed feature representations can be incorporated into deep learning. Many groups have successfully used their own deep models in applications such as detection of pulmonary nodules, detection of lymph nodes, classification of interstitial lung disease in CT images, detection of cerebral microbleeds, and detection of multiple sclerosis lesions in MR images. Notably, most of the methods described in the literature exploited deep convolutional models to maximally utilize structural information in two, two-and-a-half, or three dimensions.

Ciompi et al. (43) used a pretrained OverFeat (111) out of the box as a feature extractor and empirically showed that a CNN learned from a completely different domain of natural images can provide useful feature descriptions for classification of pulmonary perifissural nodules. Roth et al. (40) focused on training deep models from scratch. To confront the problem of data insufficiency in training deep CNNs, they expanded their data set by scaling, translation, and rotation in random overtraining samples. They augmented the test samples in a similar way; obtained CNN outputs for every augmented test sample; and took the average of the outputs of the randomly transformed, scaled, and rotated patches for detection of lymph nodes and colonic polyps. To better utilize volumetric information in images, both Ciompi et al. (43) and Roth et al. (40) considered two-and-a-half-dimensional (2.5D) information with 2D patches of three orthogonal views (axial, sagittal, and coronal). Setio et al. (42) considered three sets of orthogonal views for a total of nine views from a 3D patch and used ensemble methods to fuse information from different views for detection of pulmonary nodules.

Gao et al. (112) focused on the holistic classification of CT patterns for interstitial lung disease by using a deep CNN. They borrowed the network architecture from Reference 113, with six units at the output layer, to classify patches into normal, emphysema, ground glass, fibrosis, micronodules, and consolidation. To overcome the overfitting problem, they utilized a data augmentation strategy by generating images by randomly jittering and cropping 10 subimages per original CT slice. At the testing stage, they generated 10 jittered images and fed them into the trained CNN. Finally, they predicted the input slice by aggregation, similar to the research by Roth et al. (40).

Shin et al. (45) conducted experiments on data sets of thoraco-abdominal lymph node detection and interstitial lung disease classification to explore how the performance of a CNN changes according to architecture, data set characteristics, and transfer learning. They considered five deep CNNs, namely CifarNet (114), AlexNet (113), OverFeat (111), VGG-16 (115), and GoogLeNet (116), which achieved state-of-the-art performance in various computer vision applications. From their extensive experiments, these authors drew some interesting conclusions: (a) It was consistently beneficial for CADe problems to transfer features learned from the large-scale annotated natural image data sets (ImageNet), and (b) applications of off-the-shelf deep CNN features to CADe problems could be improved by exploring the performance-complementary properties of human-designed features.

Unlike the studies above, which used deterministic deep architectures, van Tulder & de Bruijne (35) exploited a deep generative model with a convolutional RBM as the basic building block for classification of interstitial lung disease. Specifically, they used a discriminative RBM with an additional label layer along with input and hidden layers to improve the discriminative power of learned feature representations. These experiments demonstrated the advantages of combining generative and discriminative learning objectives by achieving higher performance than that of purely generative or discriminative learning methods.

Pereira et al. (34) investigated brain tumor segmentation by using CNNs in MR images. They explored small-sized kernels in order to have fewer parameters but deeper architectures. They trained different CNN architectures for low- and high-grade tumors and validated their method in the 2013 Brain Tumor Segmentation (BRATS) Challenge,⁷ where their technique ranked at the top for the complete, core, and enhancing regions for the challenge data set. Brosch et al. (49) applied deep learning for multiple sclerosis lesion segmentation on MR images. Their model was a 3D CNN composed of two interconnected pathways, namely a convolutional pathway that learned hierarchical feature representations similar to those of other CNNs and a deconvolutional pathway consisting of deconvolutional and unpooling layers with shortcut connections to the corresponding convolutional layers. The deconvolutional layers were designed to calculate abstract segmentation features from the features represented by each convolutional layer and the activation of the previous deconvolutional layer, if applicable. In comparison to five publicly available methods for multiple sclerosis lesion segmentation, this method achieved the best performance in terms of Dice similarity coefficient, absolution volume difference, and lesion false-positive rate.

An important limitation of typical deep CNNs arises from the fixed architecture of the models themselves. When an input observation is larger than the unit in the input layer, the straightforward solution is to apply a sliding-window strategy. However, it is computationally very expensive and time/memory consuming to do so. Because of this scalability issue in CNNs, Dou et al. (36) devised a 3D fully connected network by transforming units in the fully connected layers into a 3D (1×1×1) convolutionable kernel that enabled an arbitrary-sized input to be processed efficiently (101). The output of this 3D fully connected network could be remapped back onto the original input, making it possible to interpret the network output more intuitively. For detection of cerebral microbleeds in MR images, these authors designed a cascade framework. They first screened the input with the proposed 3D fully connected network to retrieve candidates with high probabilities of being cerebral microbleeds, and then applied a 3D CNN discrimination model for final detection. These experiments validated the effectiveness of the method by removing massive redundant computations and dramatically speeding up the detection process.

3.5. Deep Learning for Computer-Aided Diagnosis

CADx provides a second objective opinion regarding assessment of a disease from image-based information. The major applications of CADx involve the discrimination of malignant from benign lesions and the identification of certain diseases from one or more images. Conventionally, most CADx systems were developed to use human-designed features engineered by domain experts. Recently, deep learning methods have been successfully applied to CADx systems.

Cheng et al. (39) used an SAE with a denoising technique (SDAE) to differentiate breast ultrasound lesions and lung CT nodules. Specifically, the image regions of interest (ROIs) were first resized to 28×28, where all of the pixels in each patch were treated as the input to the SDAE. During the pretraining step, the authors corrupted the input patches with random noise to enhance the noise tolerance of their model. Later, during the fine-tuning step, they included the resized scale factors of the two ROI dimensions and the aspect ratios of the original ROIs to preserve the original information. Shen et al. (41) created a hierarchical learning framework with a multiscale CNN to capture various sizes of lung nodules. In this CNN architecture, three CNNs that took nodule patches from different scales as input were assembled in parallel. To reduce overfitting, the authors set the parameters of the three CNNs to be shared during training. The activations of the top hidden layer in three CNNs, one for each scale, were concatenated to form a feature vector. For classification, the authors used an SVM with a radial basis function kernel and random forest, which was trained to minimize so-called companion objectives, defined as the combination of overall hinge loss function and sum of the companion hinge loss functions (117).

Suk et al. (31) used an SAE to identify Alzheimer disease or mild cognitive impairment by fusing neuroimaging and biological features. They extracted GM volume features from MR images, regional mean intensity values from PET images, and three biological features (Aβ₄₂, p-tau, and t-tau) from CSF. After training modality-specific SAEs, for each modality they constructed an augmented feature vector by concatenating the original features with the outputs of the top hidden layer of the respective SAEs. A multikernel SVM (118) was trained for clinical decision making. The same authors extended their research to find hierarchical feature representations by combining heterogeneous modalities during feature representation learning, rather than in the classifier learning step (29). Specifically, they used a DBM to find a latent hierarchical feature representation from a 3D patch, and then devised a systematic method for a joint feature representation (Figure 9a) from the paired patches of MRI and PET with a multimodal DBM. To enhance diagnostic performance, they also used a discriminative DBM by adding a discriminative RBM (119) on top of the highest hidden layer. That is, the top hidden layer was connected to both the lower hidden layer and the additional label layer that indicated the label of the input patches (Figure 9a). Using this method, the authors trained a multimodal DBM to discover hierarchical and discriminative feature representations by integrating the process of discovering features of inputs with their use in classification. Figure 9b,c shows the learned connection weights from the MRI pathway and the PET pathway.

Plis et al. (120) applied a DBN to MR images and validated the feasibility of the application by investigating whether a building block of deep generative models was competitive with independent component analysis, the most widely used method for functional MRI (fMRI) analysis. They also examined the effect of the depth of deep models in the analysis of structural MR images of a schizophrenia data set and a Huntington disease data set. Inspired by the work of Plis et al., Kim et al. (121) and Suk et al. (33) independently studied applications of deep learning for fMRI-based brain disease diagnosis. Kim et al. used an SAE for whole-brain resting-state functional connectivity pattern representation for the diagnosis of schizophrenia and the identification of aberrant functional connectivity patterns associated with schizophrenia. They first computed Pearson's correlation coefficients between pairs of 116 regions on the basis of their regional mean blood oxygenation level–dependent (BOLD) signals. After performing Fisher's r-to-z transformation on the coefficients and Gaussian normalization sequentially, they fed the pseudo-z-scored levels into their SAE. More recently, Suk et al. (33) proposed a novel framework of fusing deep learning with a hidden Markov model (HMM) for functional dynamics estimation in resting-state fMRI and successfully used this framework for the diagnosis of mild cognitive impairment (MCI). Specifically, they devised a deep auto-encoder (DAE) by stacking multiple RBMs in order to discover hierarchical nonlinear functional relations among brain regions. Figure 10 shows examples of the learned connection weights in the form of functional networks. This DAE was used to transform the regional mean BOLD signals into an embedding space, whose bases were understood as complex functional networks. After embedding functional signals, Suk et al. then used the HMM to estimate the dynamic characteristics of functional networks inherent in resting-state fMRI via internal states, which could be inferred statistically from observations. By building a generative model with an HMM, they estimated the likelihood of the input features of resting-state fMRI as belonging to the corresponding status (i.e., MCI or normal healthy control), then used this information to determine the clinical label of a test subject.

Other studies have used CNNs to diagnose brain disease. Brosch et al. (47) performed manifold learning from downsampled MR images by using a deep generative model composed of three convolutional RBMs and two RBM layers. To speed up the calculation of convolutions, the computational bottleneck of the training algorithm, they performed training in the frequency domain. By generating volume samples from their deep generative model, they validated the effectiveness of deep learning for manifold embedding with no explicitly defined similarity measure or proximity graph. Li et al. (44) constructed a three-layer CNN with two convolutional layers and one fully connected layer. They proposed to use CNNs to integrate multimodal neuroimaging data by designing a 3D CNN architecture that received one volumetric MRI patch as input and another volumetric PET patch as output. When trained end to end on subjects with both data modalities, the network captured the nonlinear relationship between two modalities. These experiments demonstrated that PET data could be predicted and estimated, given the input MRI data, and the authors quantitatively evaluated the proposed data completion method by comparing the classification results according to the predicted and actual PET images.

4. CONCLUSION

Computational modeling for medical image analysis has had a significant impact on both clinical applications and scientific research. Recent progress in deep learning has shed new light on medical image analysis by enabling the discovery of morphological and/or textural patterns in images solely from data. Deep learning methods have achieved state-of-the-art performance across different medical applications; however, there is still room for improvement. First, as witnessed in computer vision, in which breakthrough improvements were achieved by use of large numbers of training data [e.g., more than one million annotated images in ImageNet (24)], a large, publicly available data set of medical images from which deep models can find more generalized features would lead to improved performance. Second, although data-driven feature representations, especially in an unsupervised manner, have helped enhance accuracy, it would be desirable to devise a new methodological architecture involving domain-specific knowledge. Third, it is necessary to develop algorithmic techniques to efficiently handle images acquired with different scanning protocols so that it would not be necessary to train modality-specific deep models. Finally, when using deep learning to investigate underlying patterns in images such as fMRI, because of the black box–like characteristics of deep models, it remains challenging to understand and interpret the learned models intuitively.

disclosure statement

The authors are not aware of any affiliations, memberships, funding, or financial holdings that might be perceived as affecting the objectivity of this review.