1. INTRODUCTION

As described by Marks & Dunkin (1), endoscopy has evolved over many generations since 1806, when Philipp Bozzini created a light conductor to visually inspect the bladder and rectum by using a concave mirror that reflected the light of a candle. Marks & Dunkin provide a brief history of endoscopy, from a simple tube with lenses and a light source to today's methods incorporating various surgical endoscopic platforms and technology.

In 1877, more than 70 years after Bozzini's invention, Maximilian Nitze devised a cystoscope/photo-endoscope that combined lenses and electric light to examine the bladder. In 1881, Johann von Mikulicz-Radecki performed a stomach examination using a gastroscope, which was the first instrument to have an integrated miniature light bulb at its distal tip (2). In 1887, using this instrument, Gustav Killian performed the first bronchoscopy, enhancing the illumination by using a small head mirror. In 1901, using Nitze's cystoscope and without pneumoperitoneum, Hans Jacobaeus performed the first celioscopy; Jacobaeus has been credited with many publications on endoscopic explorations of the abdomen and the thorax, including papers on laparoscopy in 1912 and on thoracoscopy in 1910 (3).

Endoscopy was restricted to a small number of enthusiasts until 1932, when Georg Wolf, a German manufacturer of rigid endoscopes, produced a semiflexible gastroscope. In 1945, Karl Storz began manufacturing endoscopic devices for ear, nose, and throat surgeons (4). In 1952, the British surgeon Harold Hopkins used cold light to illuminate the endoscopic field, given that warm light can damage the canals and cavities of the body; he also invented the fibroscope (5). In 1957, inspired by Hopkins's endoscope, Basil Hirschowitz, an academic gastroenterologist best known in the field for having invented an improved optical fiber, created the first flexible fiber optic endoscope (6), in collaboration with Larry Curtiss and C. Wilbur Peters. Thousands of glass fibers integrated with flexible endoscopes are essential for endoscopic field illumination and image transmission for visualizing surgical procedures. Following advances in camera and video technologies, digital video endoscopy was created through the use of charge-coupled device image sensors in 1986, which promoted the development of modern endoscopy.

1.1. Motivation for Endoscopic Imaging

Modern endoscopy generally consists of intraoperative imaging systems and endoscopes (Figure 1). An endoscopic imaging system, such as the VISERA ELITE Platform (Olympus Corporation, Japan), usually contains a monitor, a video system center, and various light sources for different examination purposes. A typical endoscope comprises a light guide connector and tube, control body, insertion tube, and bending section with internal instrument channels. The endoscope's distal tip is generally integrated with working channels, light guides with illumination fibers, and video cameras with objective lenses. Various types of endoscopes (e.g., bronchoscope, gastroscope, colonoscope, laparoscope) can be inserted into the body through different natural orifices (e.g., mouth, nose, anus).

Endoscopic imaging is motivated by interventional diagnosis and treatment of a variety of different diseases and abnormalities that may otherwise go undetected. It provides on-site visualization of the operating field to assist surgeons in manipulating endoscopes and other surgical instruments to regions of interest during endoscopic interventions (Figure 2).

Biopsy is a diagnostic test typically associated with cancer detection and staging. It is commonly performed by surgeons or interventional radiologists to extract sample cells or tissue for pathological assessment in order to help diagnose or identify areas of concern and determine whether or not tissues are cancerous. Types of biopsies available for clinical applications include needle biopsy, skin biopsy, and bone biopsy. Most of these procedures employ a sharp tool to remove a small amount of tissue from an area of concern. The surgeon usually selects the biopsy type in accordance with the condition and the area of the patient that requires closer examination.

Endoscopic biopsies are transluminal operations employed to reach target cells or tissues inside the body in order to collect samples from tubular anatomical structures, such as the bladder, colon, or airway tree. Such biopsies use endoscopes’ working channels to accommodate and guide surgical instruments to the approximate areas of concern, where sample tissues are removed from the body. Surgeons commonly perform endoscopic biopsies to diagnose or determine the presence or extent of various diseases, such as lung, esophageal, colorectal, or breast cancer.

Although endoscopic biopsies are widely used in clinical applications, their diagnostic yield depends on the precise localization of the areas of concern from which samples are acquired. In most endoscopic biopsies, suspicious tissues cannot be observed in endoscopic imaging, because they are usually hidden beyond the surfaces of tubular structures, suggesting that surgeons can use only preoperative anatomical information and their surgical knowledge skills to blindly puncture suspicious tissues in the operating room.

Once a diagnosis of cancer is confirmed, staging and treatment will follow. Various primary strategies of cancer treatment generally involve surgery, radiotherapy, and targeted therapy, which can be employed separately or in combination. Endoscopic treatments are involved with both surgery and therapy. Whereas endoscopic surgery, also commonly known as minimally invasive surgery, typically refers to a surgical resection operation that directly removes cancerous tumors from the body, endoscopic therapy uses radiation or drug injection to kill tumor tissue in situ. Endoscopic therapy commonly employs laser ablation, which destroys problematic regions including precancerous and cancerous tissue at high temperatures in a matter of seconds.

Treatment options are usually determined by the types and stages of cancer. For cancers that are not diffuse, endoscopic surgery is the most common noninvasive or minimally invasive treatment; for example, surgical removal is most curative for colorectal cancer. Lung cancer is generally categorized into small cell or non–small cell for the purposes of treatment. While patients diagnosed with advanced-stage non–small cell lung cancer are usually treated with chemotherapy or targeted drugs, endoscopic resection and laser ablation are usually the treatments of choice for early-stage non–small cell lung cancers.

Regardless of whether the surgeon chooses surgical resection or laser ablation for treatment, the problematic tissue must be precisely targeted to obtain the optimal treatment outcome. The targeted region should be accurately associated with the localization of surgical instruments inside the patient. Accurate positioning of targeted regions and surgical instruments is therefore crucial for surgeons to perform successful endoscopic treatment. However, it remains a challenge to spatially and temporarily determine those positions in various endoscopic procedures, particularly when the targets are outside the field of view of a rigid or flexible endoscope.

Endoscopic imaging is a widely used modality that enables screening, surveillance, diagnosis, and treatment of a wide range of diseases and disorders in a noninvasive or minimally invasive way. Endoscopic imaging enables endoscopic diagnosis and treatment such as biopsy, tumor resection, and laser ablation to be easily performed when anatomical targets and surgical tools inside the body can be visualized in endoscopic images. However, in most cases, such targets cannot be observed in endoscopic views or surgical fields during interventional endoscopy. This raises two fundamental issues: (a) where to go and (b) how to get there using various endoscopic procedures or interventions, where surgeons expect clear and intuitive visualization with precise and real-time localization of areas of concern and surgical instruments. These issues provide the motivation for researchers to develop various advanced endoscopic navigation (AEN) systems.

1.2. Navigation in Endoscopy

AEN represents various surgical concepts and approaches that use computer and information technologies for surgical planning and for guiding or performing interventional diagnosis and treatment. In endoscopic diagnosis and treatment, navigation is defined by two major questions posed from different clinical applications: Where are the anatomical targets, and how do surgeons safely and quickly reach them? This definition implies that navigation can accurately identify the position of anatomical targets and simultaneously enable surgeons to automatically learn where they are and how they should orient surgical instruments associated with the targets during endoscopic interventions.

AEN is being developed to address these important questions concerning surgical instruments and anatomical targets, and it is also a leading factor in the development of robot-assisted surgery. While AEN systems are promising, they have to process a variety of big data in different modalities, as discussed in Section 2.

2. SURGICAL BIG DATA

Medical diagnosis and treatment procedures commonly involve various modalities of surgical data. During the past decade, the volume of surgical data has increased tremendously, bringing us to the era of surgical big data. AEN systems involve surgical big data that are generally classified into three main categories: (a) preoperative imaging, (b) intraoperative imaging, and (c) external sensing. Each of these categories is discussed in the following subsections.

2.1. Preoperative Imaging

Preoperative imaging is a diagnostic technology that employs various specialized scanners to acquire digital data of anatomical structures in the body. Preoperative images are used to visually diagnose diseases and abnormalities, such as suspicious tissue or tumor changes, prior to an intervention. Similar scanners are also used to collect postoperative images to evaluate surgical performance and outcomes following treatment.

Various preoperative modalities are commonly used in diagnosis and surgical planning. Computed tomography (CT) and magnetic resonance (MR) imaging techniques are frequently employed in surgical navigation systems. Diffusion tensor imaging (DTI) is a relatively new modality that uses the diffusion of water molecules to enhance contrast in MR images so as to visualize the location, orientation, and anisotropy of the brain's white matter tracts (7). More recently, diffusion spectrum imaging (DSI) was developed to address the challenge of DTI-based tractography's inability to directly image multiple fiber orientations within a single voxel (8). Since DSI helps describe regions of white matter pathways, surgeons anticipate that it could eventually be employed to guide neuroendoscopy for accurate brain tumor resection (9).

Positron emission tomography (PET) imaging is rapidly increasing in popularity and has had a significant impact on patient management and survival outcomes, such as improving surgical treatment for lung cancer with avoiding inadvertent injury and guiding surgical resection for colorectal cancer (10, 11). Whereas CT and MR are structural modalities, the integration of PET with CT or MR allows anatomic and metabolic information to be measured simultaneously. PET-CT- and PET-MR-guided interventions are increasingly being employed in cancer diagnosis and treatment.

2.2. Intraoperative Imaging

Intraoperative imaging allows surgeons to capture real-time views of the organ being operated on, as well as its anatomical surroundings, and enables more precise targeting during interventional procedures. Intraoperative imaging modalities are widely used to examine anatomical structures either on the surface of an organ or beneath it. Common intraoperative imaging modalities are discussed in the following subsections.

Optical endoscopic imaging is indispensable for most minimally invasive surgical procedures and provides surgeons with continuous and direct real-time visualization of the surgical field, as well as intuitive manipulation of surgical tools. However, it also suffers from several bottlenecks, such as a relatively limited light source and field of view, rendering it incapable of inspecting anatomical structures outside tubular organs such as the esophagus or colon. Moreover, it is unable to visualize many useful details, such as neurovascular bundles and bleeding regions on the organ surface.

Cone beam computed tomography (CBCT), as its name implies, consists of a cone-shaped X-ray beam, generated by the X-ray source and detector (image intensifier or flat panel detector), that rotates around a field of interest and captures a cylindrical volume of data (12). While conventional CT forms a fan-shaped beam and is usually used for preoperative diagnostic imaging, CBCT is often employed in the operating room, particularly in craniofacial or maxillofacial imaging in dental surgery (13). Real-time and accurate registration between CBCT and endoscopic videos can guide skull base surgery (14), and CBCT integrated with angiographic imaging is a powerful technique for intraoperative localization of cerebral arteriovenous malformations (15). Currently, several commercial systems of CBCT, such as DynaCT (Siemens Medical Solutions, Germany), XperCT (Philips Medical Systems, Netherlands), and Innova CT (GE Healthcare, United States), are available for use in clinical applications.

Ultrasound (US) imaging uses high-frequency sound waves to image soft tissue, enabling physicians to evaluate, diagnose, and treat medical conditions without the risk associated with exposure to ionizing radiation. A major advantage of this technique is its ability to capture images in real time, showing the motion of the organs as well as blood flowing through the blood vessels. Recently, ultrafast US, a new technology with frame rates typically faster than 1,000 frames/s, has been used for deep superresolution vascular imaging (16).

Endoscopic US (EUS) enables surgeons to image the interior structures and surroundings of anatomical organs during interventional endoscopy. The endoscope uses either a US transducer fixed to its distal tip (e.g., endobronchial US) or an ultrathin radial ultrasonic probe through its working channel (Figure 3). EUS is a fast-developing surgical area associated with advances in technology, resolution, and instrumentation, and it is increasingly being extended to applications in specialties such as laparoscopic resection for gastric tumors (17) and diagnosis and staging of lung cancer (18).

Intraoperative MR imaging is a relatively new modality that allows surgeons to monitor a surgical site using MR during surgery. Such a modality is most often employed in conjunction with neurosurgery, particularly with endoscopic transsphenoidal surgery for pituitary adenoma resection (19, 20), where it provides the surgeon with the ability to confirm fenestrations and biopsies, detect complications, and redefine anatomical changes during the operation. More recently, a study combining intraoperative MR imaging with neuronavigation demonstrated that such an imaging technique can improve the surgical outcome of endoscopic transsphenoidal surgery (21).

Optical coherence tomography (OCT) is a new imaging technology that uses low-coherence interferometry to capture two-dimensional (2D) micrometer-resolution images of optical scattering from internal tissue microstructures (22, 23). It provides a depth-resolved, noninvasive, nondestructive imaging modality similar to US imaging (24). OCT is commonly used for diagnosis and surgery of eye disease (25–27). 2D OCT images can be reconstructed to provide three-dimensional (3D) visualization (Figure 4). OCT is currently being applied to various clinical fields to examine tubular anatomical structures using different transluminal tools such as endoscopes, needles, and other imaging probes (28). In addition, OCT is being extended to noninvasive depth-resolved functional imaging that offers spectroscopic, polarization-sensitive, blood flow, and physiological tissue information. These extensions have the potential to improve image contrast while simultaneously enhancing pathologies by using localized metabolic properties or physiological states (28). A thorough survey of OCT and its medical applications is available elsewhere (29).

Single-photon emission computed tomography (SPECT) is a functional imaging technique that uses γ cameras or probes to detect γ-rays emitted by an injected radioactive substance or tracer to acquire multiple 2D projections from various angles (30). On the basis of tomographic reconstruction algorithms, the multiple projections are reconstructed into 3D images (31). SPECT provides functional information similar to that obtained from PET about blood flow to tissues and metabolism, but it enables real-time in vivo imaging of several γ radioactive compounds in the body during intervention.

SPECT imaging is used in many clinical situations (32–34), particularly for coronary disease (35–37). More recently, SPECT has been employed to create a commercialized image-guided system, declipseSPECT (SurgicEye GmbH, Germany), that provides 3D breast imaging, navigation, and control of complete resection. During laparoscopy, the declipseSPECT system uses freehand SPECT technology to generate 3D images of radioactively marked structures and provides surgeons with an intraoperative 3D imaging system for precise, minimally invasive sentinel lymph node biopsy.

2.3. External Sensing

External sensing refers to the use of external devices that usually do not form part of an endoscope to track surgical instruments. On the basis of real-time sensing or tracking information, the six-degrees-of-freedom (6DoF) position and orientation of surgical tools used in intervention can be associated with preoperative computational anatomical models. Currently, various external sensing techniques (see Section 3.2.2) are available for surgical navigation.

3. METHODOLOGY

AEN is generally recognized as an innovative concept, a measurement toolbox, and an information container that provides surgeons with the appropriate information at the right place and the right time during interventional endoscopy. Undoubtedly, surgical big data play a vital role in exploring a variety of AEN systems. Whereas preoperative image processing is employed to create computational models of patient anatomy, intraoperative data analysis provides surgeons with direct visualization of the surgical field. The general principles of various AEN systems are computational anatomy, surgical navigation, intuitive visualization, and interactive software (Figure 5). These principles are discussed in the following subsections.

3.1. Computational Anatomy

Computational anatomy is a relatively new discipline that uses various imaging modalities, particularly preoperative imaging, to comprehensively describe human anatomy in a digital format (38) and create precise virtual models of anatomical structures and organs. The development of accurate virtualized models is paramount for surgical navigation systems, since such models provide maps and target localization during surgery. Generally, computational anatomy associated with various modalities and algorithms aims to answer the question of where the anatomical targets are prior to intervention.

Conventionally, surgeons form mental images of the organ and anatomical structures of areas of concern from a 3D volume or 2D images prior to surgery, and they are trained to interpret these images in relation to the 3D surgical field during the procedure. Segmentation and registration are indispensable computational anatomy techniques to process and analyze medical images and reduce the surgeon's cognitive load. Segmentation detects and extracts target boundaries or regions of interest within 2D slices or 3D volumes in multiple modalities and is commonly classified into manual, interactive, and automatic approaches. During the last two decades, many segmentation algorithms have been developed (39, 40). Registration spatially aligns reference and target images from either the same modality or different modalities so that relevant information in each modality can be optimally integrated or compared (41). A recent retrospective view of medical image registration over the past two decades is provided by Viergever et al. (42).

While computational anatomy promotes diagnostic, preoperative planning (Figure 6), and surgical simulation (Figure 7), it remains challenging to explore advanced methodologies for precise and robust retrieval of anatomical structure information from different types of medical images, which is the virtual equivalent of dissecting a real human body. This challenge arises mainly from clinical variations pertaining to patient differences, partial volume effects, and various levels of imaging resolution. Recently, Schork (43) reported a new concept of personalized (or precision) medicine. Moreover, machine intelligence and learning technologies could provide a powerful tool to address the challenges in computational anatomy (44, 45). Additionally, every year many new medical image computing approaches are published in two flagship journals, Medical Image Analysis and IEEE Transactions on Medical Imaging.

3.2. Surgical Navigation

In image-guided endoscopic surgery, the guidance of an instrument toward a desired target is typically defined as surgical navigation. In this respect, surgical navigation or active surgical guidance is the most important element of AEN systems. Surgical navigation can be described as a combination of computational anatomy, tracking algorithms or devices, image data confluence, and specialized instruments to assist and guide surgeons during intervention. It provides accurate real-time positioning of in vivo anatomical structures and organs as well as surgical instruments overlaid on preoperative images in the operating room. The general principles of surgical navigation methods are discussed below.

Vision-based tracking is a navigation method used to register 2D endoscopic video images to preoperative 3D data in real time. The 3D data are usually rendered as 2D virtual images that correspond to various endoscopic camera 6DoF poses, including position and orientation parameters, in the coordinate system of the 3D preoperative computational anatomy (Figure 8). We also refer to this tracking method as video–volume (2D–3D) registration, which can be formulated by an optimization procedure:

1.

where ^AMⁱ_C is the optimal camera pose predicted at the ith endoscopic image I_i and indicates the transformation from endoscopic camera C to computational anatomy A, is the similarity function, and I_j(·) is a 2D virtual rendering procedure that corresponds to camera pose ^AM^j_C at the jth iteration in optimization. Although can also be defined as a dissimilarity function, Equation 1 then becomes a minimization procedure to estimate ^AMⁱ_C.

Many authors have discussed video-based tracking. Deguchi et al. (46) proposed a selective image similarity measure to register endoscopic video sequences and preoperative volumes, which was improved by Luo et al. (47), while Merritt et al. (48) reported an interactive CT–video registration technique to continuously guide bronchoscopic intervention. Mirota et al. (14) used high-accuracy 3D image–based registration to align endoscopic video and CBCT images, and a video–volume registration method was developed by Luo & Mori (49) by use of discriminative structural similarity measure to track endoscope motion tracking. Shen et al. (50) explored a depth reconstruction method to achieve a similar goal, and Zhang et al. (51) employed a 3D graph–based optimization method for simultaneous registration of position and orientation during intravascular US intervention.

External tracking refers to the use of devices and systems to localize surgical instruments in real time during endoscopic surgery. This tracking technology typically uses external position sensors, such as electromagnetic (EM) sensors that are attached to the surgical tools to measure their movement, to determine ^AMⁱ_C as follows:

2.

where ^AM_E, ^EMⁱ_S, and ^SM_C are transformation matrices describing the spatial relationships between computational anatomy A, external tracking system E, external sensor S, and endoscopic camera C, and calculated by the methods discussed in the following subsections.

AEN systems generally employ a combination of endoscopes, EUS probes, and external trackers with position sensors. These integrated devices provide different sensing information in different coordinate systems. In order to relate this information to a common reference frame, calibration must be performed prior to navigation.

The camera calibration process employs employs standard algorithms and images of a special pattern (e.g., chessboard or square grid) to estimate the camera's intrinsic parameters (focal length, skew, distortion, and image center) (52). On the basis of these parameters, camera distortion on endoscopic video images can be corrected. This is particularly important for vision-based tracking methods because it improves the tracking accuracy during navigation. Zhang (53) proposed a flexible new technique for camera calibration that is now used in many computer vision tasks. Hartley & Kang (54) simultaneously calibrated a camera's radial distortion function along with the other internal calibration parameters. The advantage of this method is that it determines radial distortion in a parameter-free manner without the need to use any particular model. More recently, a unified model has been reported to calibrate a wide variety of camera models, such as pinhole, fisheye, catadioptric, and multicamera networks (55).

Hand–eye calibration (HEC) aims to determine the relationship ^SM_C between external position sensors (hand) and imaging devices (eye), such as endoscopic cameras and EUS probes. The HEC problem originally arose in the area of robotics, and several classical HEC methods (56–59) are still in routine use. Most HEC approaches usually employ the results of internal and external parameters from camera calibration using specific patterns. More recently, camera calibration–free approaches have been invoked to solve the HEC problem (60, 61).

Initial registration is the process of determining the spatial relationship ^AM_E between computational anatomy and external tracking system prior to real-time navigation. This procedure, also referred to as tracker-to-model registration, aims to determine the spatial transformation ^AM_E in Equation 2.

Two strategies of marker-based and marker-free registration are commonly used to compute the optimal solution ^AM_E. Marker-based registration employs either artificial fiducial markers placed on the body or natural/anatomical fiducial markers available within the body or on its surface. Wognum et al. (62) validated a deformable image registration algorithm with 30–40 fiducial markers for pelvic cancer surgery, and Hughes-Hallett et al. (63) reviewed a fiducial-based registration method for guided partial nephrectomy. Inoue et al. (64) improved the accuracy of the point-based rigid-body registration algorithm with implanted fiducial markers for breast intervention, while Tabrizi & Mahvash (65) used five fiducial markers to perform initial registration during image-guided neurosurgery. Marker-free registration does not require any artificial or natural fiducial markers but instead employs the constraints of typical anatomical structures to estimate ^AM_E. Klein et al. (66) proposed a fiducial marker–free method by maximizing the percentage of external sensor measurements inside the preoperative volume to predict ^AM_E. Deguchi et al. (67) explored a marker-free framework that minimizes the distance between external sensor outputs and the center line of the organ, and an initial registration strategy to estimate the relationship ^AM_E without any fiducial markers was reported by Hofstad et al. (68). Luo (69), Luo & Mori (70), and Luo et al. (71) developed several marker-free registration methods to calculate ^AM_E.

Optical tracking uses an external position sensor to perceive IR-emitting or retro-reflective markers affixed to a surgical tool or object. The position sensor determines the tool's position and orientation in accordance with the information that the sensor receives from such markers, which are generally classified into active and passive categories. An example of a typical tracking system is the Polaris device (Northern Digital Inc., Canada); however, this device can be used only with rigid endoscopes, where the markers are fixed at the end distal to the camera.

EM tracking systems use embedded sensor coils to determine the location of objects. Each system consists of three main components: a control unit, EM sensors, and an EM transmitter that establishes a tracking volume. When the object is located inside the tracking volume, currents are induced in the coils. These currents are used to compute the position and orientation of the object in real time.

Existing EM tracking systems include the 3D Guidance product suite (Ascension Technology Corporation, United States) and Aurora (Northern Digital Inc., Canada). In contrast to the Polaris device, these systems require no line-of-sight constraints from the sensor (tool) to the field generator and can be embedded in nonrigid endoscopes. Today's EM trackers are widely used in various surgical navigation systems. Although optical and EM tracking are routinely employed to minimally invasive surgical procedures, there are a number of alternative trackers available as well, as discussed below.

Stereoscopic vision is employed to perceive depth information and 3D structure derived from video information from two or more video cameras. It is a powerful technique to estimate the position and orientation of objects within a visual scene. MicronTracker (ClaroNav, Canada) uses stereoscopic vision to create a new generation of trackers that can detect and track specially marked objects. This external tracking system employs visible light and computer vision to detect fully passive marked targets and track them by processing standard video images. ClaroNav reported that MicronTracker can be employed in various surgical procedures, including image-guided intervention, ablation, and biopsy; can be operated manually or using robotics; and can assist augmented-reality (AR) procedures by providing direct visualization (72).

Inertial tracking uses a miniature microelectromechanical triaxial inertial sensor attached at the endoscope's tip to measure the impact of gravity on each of the three orthogonal accelerometer axes (73). Similar to EM sensors, inertial sensors are very small and can be unobtrusively used for endoscopic image reorientation. However, inertial tracking can measure only relative changes in pose, rather than determining absolute values.

An optical position sensor is a microscopic image acquisition device that uses an array of photodiodes to convert light into an electrical current and perceive the position of a light spot. The sensor, including an optical lens, light source, and digital signal processor, can measure motion relative to an object's surface. Such a system acquires sequential surface images that are processed to determine 2D displacements of the surface.

On the basis of optical position sensor techniques, a new external tracking prototype can be created to track the endoscope's movement. This tracking prototype has been demonstrated to be an effective strategy to navigate flexible endoscopes (74).

Radio-frequency identification (RFID) has recently been developed to localize RFID-tagged objects with millimeter accuracy using phase difference (75, 76). This technology shows great promise for clinical applications because of its wireless, extremely small RFID tags and inherent powerful identification ability.

The Calypso system (Varian Medical Systems, United States) is a wireless four-dimensional localization system that uses a set of three transponders generating radio-frequency waves. The system has provided surgeons with accurate alignment to a target prostate in real time, with an error of 2.0 mm or better, and assists in avoiding unnecessary radiation to healthy tissues (e.g., the bladder) during prostate radiotherapy (77, 78).

With advances in optics, optical fibers can be used to quasi-continuously detect strain and temperature along the fibers’ direction. Commercially available optical backscatter reflectometers (Luna Innovations, Roanoke, Virginia) enable inspection of seven million data points along a 70-m-long fiber, corresponding to a spatial resolution of 10 μm. The ShapeTape device (Measurand, Canada) has been commercialized to track various surgical tools. Koizumi et al. (79) proposed ShapeTape-driven tracking for 3D US systems. Li et al. (80) developed and evaluated a new body–seat interface shape measurement system based on fiber-optic tracking, and Housden et al. (81) used ShapeTape to track a 2D US probe for freehand 3D US.

Hybrid tracking combines vision-based and external tracking techniques to navigate surgical instruments. It aims to tackle the disadvantages of using either vision-based methods or external tracking alone; for example, the initial registration discussed above is a rigid procedure that can lead to inaccurate navigation caused by either tissue deformation or the inherent drawbacks of tracking devices. Hybrid tracking also refers to multimodal information fusion of preoperative data (e.g., CT or MR volume), intraoperative videos (endoscopic or US images), and external tracking measurements. Accurate and real-time fusion strategies are the key to developing hybrid navigation systems.

Numerous hybrid tracking approaches have been discussed in the literature. Feuerstein et al. (82) proposed magneto-optical tracking of flexible laparoscopic US, and Soper et al. (83) reported hybrid tracking based on Kalman filtering for bronchoscopic navigation. Hybrid localization methods have been widely discussed for robotic endoscopic capsules (84, 85). Reichl et al. (86) explored a hybrid endoscope tracking with guaranteed smooth output, while Luo and colleagues (87–91) developed several hybrid navigation methods by using stochastic filtering and evolutionary computation algorithms.

3.3. Intuitive Visualization

All preoperative and intraoperative data must be appropriately presented to surgical personnel in the operating room. AEN requires a display system that provides surgeons with critical structural and functional information in real time, and should provide minimal interruption to the surgical work flow. 3D volumetric data must be presented to the surgeon in an intuitive manner so as to provide important and understandable information to evaluate anatomical structure and function.

A major advantage of navigation is simultaneous visualization of tracked surgical instruments in relation to multimodal data. Volumetric data can be visualized in several ways. Slice-based methods usually present orthogonal slices, allowing the surgeon to view patient 3D data in the axial, sagittal, and coronal directions (Figure 9). Multiplanar reconstruction is another way to inspect volumetric data with more than one slice orientation. Volume rendering is a visualization technique that uses a transfer function to assign each voxel an opacity or color that relates to the voxel's intensity in volumetric data (92). While the human visual system typically differentiates structures of interest in volumetric data from the surrounding image data by a boundary or a material interface, surface rendering is an intuitive way to separate structures of interest and simultaneously render the surface opaque and make other tissues transparent (93). Surface rendering requires previously segmented 3D data that are further processed in an intermediate step to generate anatomical 3D models, for example, with the marching cubes algorithm (94). In this respect, volume-based methods are more amenable to automatic algorithms than surface rendering. Moreland (92) recently published a thorough survey that discusses the range of current visualization pipelines.

AR was initially developed to solve the problem of how to integrate 3D virtual objects into a 3D real environment in real time. Various applications in medicine, manufacturing, visualization, path planning, and military operations have been described by Azuma et al. (95). Mixed reality refers to a combination of real and virtual images, as defined by Milgram & Kishino's (96) taxonomy. AR techniques are widely used to enhance intraoperative vision during various surgical interventions (63, 97–100).

3.4. Interactive Software

In order to ensure optimal interaction between surgeons and AEN systems, it is important to combine pre- and intraoperative data during the intervention so that they can be visualized in an intuitive manner. Moreover, reliable software, which simultaneously implements and updates various procedures relating to anatomical computation, surgical navigation, and intuitive visualization in real time, is a critical component of the AEN system (101).

Most commercial navigation software systems are proprietary, but since their development requires intensive effort, several open-source guidance software platforms and public libraries are available for academic use and surgical navigation development. 3D Slicer (https://www.slicer.org/) is a well-known software platform for medical image informatics, image processing, and 3D visualization that runs on various operating systems such as Windows, iOS, and Linux. The Image-Guided Surgery Toolkit (IGSTK) platform (http://www.igstk.org/) is a component-based framework that provides a common functionality for image-guided surgery applications. The Medical Imaging Interaction Toolkit (MITK) (http://mitk.org/wiki/MITK) is a free platform for development of interactive medical image processing software. Note that 3D Slicer, IGSTK, and MITK all use the open-source libraries of the Visualization Toolkit (VTK) and Insight Segmentation and Registration Toolkit (ITK). ITK (https://itk.org/) is a cross-platform system that provides developers with an extensive suite of software tools for image analysis, and VTK (http://www.vtk.org/) is a freely available library for 3D computer graphics, image processing, and visualization.

4. CLINICAL APPLICATIONS

Endoscopic intervention is broadly employed to inspect and operate on organs, airways, and vessels of the body to diagnose and treat various diseases in a minimally invasive manner. Originally, endoscopy was used only in the gastrointestinal (GI) tract, including the esophagus, stomach, and colon, but today it is widely used in the head and neck, throat, lung, abdomen, urinary tract, joints, and other areas. Clinical endoscopic applications (Table 1) are discussed in the following subsections.

Table 1

Endoscopic navigation systems in clinical procedures

4.1. Neurosurgical Endoscopy

Neurosurgical endoscopic navigation usually uses optical tracking, MR data, and rigid endoscopes to provide surgeons with real-time online guidance and enhance the accuracy and safety during brain tumor resection (102). While neuroendoscopy is not a predominant method for neurosurgery, it can be employed when a target is close to a region accessible from a natural orifice. Registration between endoscopic video and C-arm CBCT is utilized to guide endoscopic skull base surgery (14). Recently, Torres-Corzo et al. (103) employed an electromagnetically navigated flexible neuroendoscope to explore the ventricles and basal cisterns of a patient with hydrocephalus.

4.2. Respiratory Endoscopy

Respiratory tract diseases can be diagnosed and treated by various endoscopic procedures. Scopis hybrid navigation (Scopis GmbH, Germany) is a surgical navigation system with AR capabilities for endoscopic sinus surgery (104). More recently, registration and fusion quantification were discussed in relation to AR-based nasal endoscopic surgery (105), and image-guided laryngoscopy was introduced as a possible alternative to conventional laryngectomy surgery (106). High-speed laryngoscopic recordings have been used to create 3D reconstructions of human laryngeal dynamics (107).

Bronchoscopy is routinely performed for lung cancer diagnosis, staging, and treatment, and various bronchoscopic navigation systems (Figure 10) have been described in the literature (47–50, 74, 83, 91). Thoracoscopy or video-assisted thoracic surgery is usually employed for pleural biopsy and pulmonary lesions in the lung and the heart (108, 109).

4.3. Gastrointestinal Endoscopy

The GI tract consists of various organs, including the esophagus, stomach, small intestine, large intestine (colon), and rectum. These organs can be inspected by different endoscopes. Esophagoscopy is a transnasal procedure performed under sedation or general anesthesia (110), whereas transorally introduced gastroscopy is an effective procedure to determine the resection margins of gastric cancer (111). Virtual gastroscopy, examination of a 3D upper GI tract reconstruction from CT scans, is used to evaluate malignancies of the stomach (112).

Colonoscopy is employed to detect and treat polyps, tumors, bleeding, or inflamed regions in the lower GI tract. However, EM-CT or colonoscopic video–CT registration and colonoscope tracking remain challenging for the advancement of colonoscopic navigation because of potential large deformations that can occur during intervention (113, 114).

4.4. Abdominal Laparoscopy

Laparoscopy is usually performed in the abdomen or pelvis, and image-guided laparoscopic procedures are frequently employed for liver surgery (115, 116). Laparoscopic prostatectomy is widely used for prostate cancer surgery (117, 118); this robot-assisted procedure is increasingly being performed for prostate tumor resection (119). Navigated laparoscopic gastrectomy is employed to treat gastric cancer (120).

4.5. Urinary Endoscopy

The urinary tract includes the urethra, kidney, ureters, and bladder. Cystoscopy plays an important role in predicting the grade and stage of bladder cancer (121, 122). Percutaneous nephroscopy is a routine surgical procedure that treats large or complex renal stones (123). Laparoscopic partial nephrectomy is an effective means of removing small renal tumors and simultaneously preserving the remainder of the kidney. An AR nephrectomy navigation system has been developed using EM tracking (124).

4.6. Joint Arthroscopy

Arthroscopy is a minimally invasive surgical procedure that inspects, diagnoses, and treats diseases inside joints such as hips and knees by use of an arthroscope. Computer-assisted arthroscopic navigation systems have been explored for hip and knee surgery (125, 126).

4.7. Others

Other surgical procedures also use endoscopes for various interventions. Cardiac surgery may use a 3D high-definition endoscopic system with augmented visualization (127), and endoscopic carpal tunnel release has been employed in hand surgery (128). Spinal surgery may use an epiduroscope to identify abnormalities in the epidural space, establish diagnosis, and administer treatment (129).

5. ENDOSCOPIC ADVANCES

Endoscopy is undergoing an evolution, and major improvements are being introduced as new technologies emerge. The concept of navigation is revolutionizing the development of modern endoscopy. In addition, new endoscopic devices, imaging, video processing, and interdisciplinary techniques are enhancing endoscopic interventions that have the potential to influence diagnosis, treatment, and clinical outcomes.

5.1. Wireless Capsule Endoscopy

The wireless capsule endoscope is a relatively new surgical device used to examine diseases in the GI tract, particularly the stomach and colon, that is completely changing conventional GI endoscopy, which must be performed manually. Current research in this area focuses on automatic detection, classification, and accurate localization of structures within the endoscopic field of view, as well as endoscopic video stabilization (130–132).

5.2. Robotic Endoscopy

An emerging technology trend in endoscopy is robotization, which aims to accurately and remotely manipulate endoscopes and other surgical instruments to targets and their surroundings. Robot-assisted laparoscopic procedures are finding increasingly broad use in abdominal procedures with the da Vinci surgical system (133, 134). Natural orifice transluminal endoscopic surgery (NOTES) is another new paradigm for laparoscopic procedures (135).

In contrast to the da Vinci robot and NOTES systems, which typically use rigid laparoscopes, the robotization of other endoscopic procedures (e.g., colonoscopy) that use flexible endoscopes presents a more challenging problem. In this respect, the combination of endoscopic navigation and robotized endoscope is a promising research direction. van der Stap et al. (136) proposed an image-based navigation strategy to robotize a flexible endoscope, and a framework comprising robotic steering and lumen centralization has been reported to automate the colonoscope (137). Shape-sensing techniques for continuum robots have been reviewed for endoscopic procedures (138).

5.3. New Endoscopic Imaging

Advanced imaging devices and technologies have the potential to greatly enhance endoscopic images. Newly available imaging devices and modalities that can be used in endoscopic interventions are as follows (139).

Optical biopsy is a relatively new surgical technology that provides surgeons with online tissue histological analysis by using the properties of light during endoscopy. The endomicroscope is a new device for optical biopsy. Confocal laser endomicroscopy (CLE) was initially proposed for diagnosis of GI disease, and a flexible version of this instrument, probe-based CLE (pCLE), is used in endoscopy. Volumetric laser endomicroscopy has demonstrated an improved diagnostic performance over that of pCLE (140).

Endocytoscopy is another optical biopsy technique that uses a high-power, fixed-focus objective lens to achieve ultrahigh magnification of GI and respiratory tract mucosa at the cellular level. However, the precise role of this technique in the GI and respiratory tracts has yet to be determined.

Near-IR (NIR) fluorescence, classified as a molecular imaging modality, is an intraoperative imaging technique that employs NIR fluorescent light to identify suspicious targets and their margins during a surgical procedure. Since normal tissue and benign and malignant tumors have different concentrations of hemoglobin and water, as well as different levels of oxygen and ultrastructural scattering, NIR fluorescence provides a novel way to quantify blood and water concentrations and evaluate structural and functional information in tissue at the surgical site.

Intraoperative NIR fluorescence imaging is a fast-developing modality that enhances contrast and depth of tissue penetration relative to visible light and offers real-time visual information during surgery (141). An open issue involves the development of algorithms to more accurately reconstruct and display NIR fluorescence images and integrate them with preoperative CT or MR images and endoscopic videos. Additionally, fluorescence- or firefly-guided endoscopic intervention (Figure 11) is a promising development in cancer diagnosis and treatment.

5.4. Endoscopic Video Analysis

Video processing may be applied to augment endoscopic visualization of the surgical field and enhance image quality. Endoscope video images suffer from several drawbacks, including limited illumination and field of view, surface information that is not apparent to the naked eye, and surgical smoke. In order to address these issues, various video processing algorithms need to be developed in the following areas:

5.5. 3D Printing Technology

3D printing is becoming increasingly important in medicine, especially in surgery (144), providing methodology that uses medical 3D data to generate 3D physical models. On the basis of these models, surgical training, planning, and simulation can be performed more accurately and effectively. Furthermore, surgeons can use these models to intuitively guide procedures in the operating room (Figure 12). However, 3D printing implementation is time-consuming and very expensive, limiting its widespread use. Another issue involves the development of automatic and seamless fusion of 3D printing models and endoscopic interventions to enhance surgical navigation during interventional endoscopy.

6. CONCLUSIONS

Interventional endoscopy plays a critical role in diagnosing, staging, and treating various diseases in a minimally invasive manner. The concept of endoscopic navigation has revolutionized conventional endoscopic interventions and has provided surgeons with more precise, efficient, and reliable means of diagnosis and treatment. This review has investigated various technical aspects of AEN and has shown that several commercial surgical navigation systems can be used clinically to improve the precision and quality of endoscopic procedures. However, endoscopic navigation is by no means mature, and advances are ongoing. The core elements of computational anatomy, surgical tracking and navigation techniques, intuitive visualization approaches, and interactive software are becoming established while simultaneously evolving for the next generation of navigated endoscopic systems (Figure 13).

SUMMARY POINTS

FUTURE ISSUES

disclosure statement

T.M.P. holds a research grant from Intuitive Surgical Inc. to study the processing of endoscopic video. The other authors are not aware of any affiliations, memberships, funding, or financial holdings that may be perceived as affecting the objectivity of this review.