AbstractÐThis paper describes a robust hierarchical algorithm for fully-automatic registration of a pair of images of the curved human retina photographed by a fundus microscope. Accurate registration is essential for mosaic synthesis, change detection, and design of computer-aided instrumentation. Central to the newalgorithm is a 12-parameter interimage transformation derived by modeling the retina as a rigid quadratic surface with unknown parameters, imaged by an uncalibrated weak perspective camera. The parameters of this model are estimated by matching vascular landmarks extracted by an algorithm that recursively traces the blood vessel structure. The parameter estimation technique, which could be generalized to other applications, is a hierarchy of models and methods: an initial match set is pruned based on a zeroth order transformation estimated as the peak of a similarity-weighted histogram; a first order, affine transformation is estimated using the reduced match set and least-median of squares; and the final, second order, 12-parameter transformation is estimated using an M-estimator initialized from the first order estimate. This hierarchy makes the algorithm robust to unmatchable image features and mismatches between features caused by large interframe motions. Before final convergence of the M-estimator, feature positions are refined and the correspondence set is enhanced using normalized sum-of-squared differences matching of regions deformed by the emerging transformation. Experiments involving 3,000 image pairs �1; 024 1; 024 pixels) from 16 different healthy eyes were performed. Starting with as low as 20 percent overlap between images, the algorithm improves its success rate exponentially and has a negligible failure rate above 67 percent overlap. The experiments also quantify the reduction in errors as the model complexities increase. Final registration errors less than a pixel are routinely achieved. The speed, accuracy, and

Abstract—Motivated by the problem of retinal image registration, this paper introduces and analyzes a new registration algorithm called Dual-Bootstrap Iterative Closest Point (Dual-Bootstrap ICP). The approach is to start from one or more initial, low-order estimates that are only accurate in small image regions, called bootstrap regions. In each bootstrap region, the algorithm iteratively: 1) refines the transformation estimate using constraints only from within the bootstrap region; 2) expands the bootstrap region; and 3) tests to see if a higher order transformation model can be used, stopping when the region expands to cover the overlap between images. Steps 1): and 3), the bootstrap steps, are governed by the covariance matrix of the estimated transformation. Estimation refinement [Step 2)] uses a novel robust version of the ICP algorithm. In registering retinal image pairs, Dual-Bootstrap ICP is initialized by automatically matching individual vascular landmarks, and it aligns images based on detected blood vessel centerlines. The resulting quadratic transformations are accurate to less than a pixel. On tests involving approximately 6000 image pairs, it successfully registered 99.5 % of the pairs containing at least one common landmark, and 100 % of the pairs containing at least one common landmark and at least 35 % image overlap. Index Terms—Iterative closest point, medical imaging, registration, retinal imaging, robust estimation.

This paper describes an algorithm to continually and accurately estimate the absolute location of a diagnostic or surgical tool (such as a laser) pointed at the human retina, from a series of image frames. We treat the problem as a registration problem, using diagnostic images to build a spatial map of the retina and then registering each on-line against this map. Since the image location where the laser strikes the retina is easily found, this registration determines the position of the laser in the global coordinate system defined by the spatial map. For each on-line image, the algorithm computes similarity invariants, locally valid despite the curved nature of the retina, from constellations of vascular landmarks. These are detected using a high-speed algorithm that iteratively traces the blood vessel structure. Invariant indexing establishes initial correspondences between landmarks from the on-line image and landmarks stored in the spatial map. Robust alignment and verification steps extend the similarity transformation computed from these initial correspondences to a global, high-order transformation. In initial experimentation, the method has achieved 100% success on 1024 1024 retina images. With a version of the tracing algorithm optimized for speed on 512 512 images, the computation time is only 51 milliseconds per image on a 900MHz Pentium III processor and a 97% success rate is achieved. The median registration error in either case is about 1 pixel.

Abstract—This paper presents a set of algorithms for robust detection of vasculature in noisy retinal video images. Three methods are studied for effective handling of outliers. The first method is based on Huber’s censored likelihood ratio test. The second is based on the use of a-trimmed test statistic. The third is based on robust model selection algorithms. All of these algorithms rely on a mathematical model for the vasculature that accounts for the expected variations in intensity/texture profile, width, orientation, scale, and imaging noise. These unknown parameters are estimated implicitly within a robust detection and estimation framework. The proposed algorithms are also useful as nonlinear vessel enhancement filters. The proposed algorithms were evaluated over carefully constructed phantom images, where the ground truth is known a priori, as well as clinically recorded images for which the ground truth was manually compiled. A comparative evaluation of the proposed approaches is presented. Collectively, these methods outperformed prior approaches based on Chaudhuri et al. (1989) matched filtering, as well as the verification methods used by prior exploratory tracing algorithms, such as the work of Can et al. (1999). The Huber censored likelihood test yielded the best overall improvement, with a 145.7 % improvement over the exploratory tracing algorithm, and a 43.7 % improvement in detection rates over the matched filter. Index Terms—Hypothesis testing, mathematical models of vasculature, retinal fundus images, robust model selection, vasculature detection and segmentation, vessel enhancement, vessel segmentation. I.

Real-time spatial referencing is an important alternative to tracking for designing spatially-aware ophthalmic instrumentation for procedures such as laser photocoagulation and perimetry. It requires independent, fast registration of each image frame from a digital video stream (1024 1024 pixels) to a spatial map of the retina. Recently, we have introduced a spatial referencing algorithm that works in three primary steps: (1) tracing the retinal vasculature to extract image feature (landmarks), (2) invariant indexing to generate hypothesized landmark correspondences and initial transformations, and (3) alignment and veri cation steps to robustly estimate a 12-parameter quadratic spatial transformation between the image frame and the map. The goal of this paper is to introduce techniques to minimize the amount of computation for successful spatial referencing. The fundamental driving idea is to make feature extraction subservient to registration and therefore only produce the information needed for veri ed, accurate transformations. To this end, the image is analyzed along one-dimensional, vertical and horizontal grid lines to produce a regular sampling of the vasculature, needed for step (3) and to initiate step (1). Tracing of the vascular is then prioritized hierarchically to quickly extract landmarks and groups (constellations) of landmarks for indexing. Finally, the tracing and spatial referencing computations are integrated so that landmark constellations found by tracing are tested immediately. The resulting implementation is an order-of-magnitude faster with the same success rate. The average total computation time is 45.5 milliseconds per image on a 2.2 GHz Pentium Xeon processor.

A new technique is presented for mosaicing sparselyoverlapping image sets, with a target application of assisting the diagnosis and treatment of retinal diseases. The geometric image transformations required to construct the mosaics are estimated by (1) estimating the transformations between as many pairs of images as possible, (2) extracting sets of constraints (correspondences) from the successfully registered image pairs, and (3) using these constraint sets to simultaneously (jointly) estimate the final transformations. Unfortunately, this may not be sufficient to construct seamless mosaics when two images overlap but can not be successfully registered (step 1). This paper presents a new method to generate constraints between such image pairs, and use these constraints to estimate a more consistent set of transformations. For each pair, transformation parameter covariance matrices are computed and used to estimate the mapping error covariance matrices for individual features from one image. These features are matched in the second image by minimizing the resulting Mahalanobis distance. The generated correspondences are validated using robust estimation techniques and used to refine the estimates. The steps of covariance computation, matching, and transform estimation are repeated for all relevant image pairs until the final alignment converges. Results are presented and evaluated for several difficult image sets to illustrate the efficacy of the techniques.

Certain surgical procedures require a high degree of precise manual control within a very restricted area. Retinal surgeries are part of this group of procedures. During vitreoretinal surgery, the surgeon must visualize, using a microscope, an area spanning a few hundreds of microns in diameter and manually correct the potential pathology using direct contact, free hand techniques. In addition, the surgeon must find an effective compromise between magnification, depth perception, field of view, and clarity of view. Pre-operative images are used to locate interventional targets, and also to assess and plan the surgical procedure. This paper proposes a method of fusing information contained in pre-operative imagery, such as fundus and OCT images, with intra-operative video to increase accuracy in finding the target areas. We describe methods for maintaining, in real-time, registration with anatomical features and target areas using image processing. This registration allows us to produce information enhanced displays that ensure that the retinal surgeon is always in visual contact with his/her area of interest. 1.

Automated image analysis algorithms are presented for detection and classification of changes in longitudinal time-series of color retinal fundus images. They are applicable to clinical practice, quantitative scoring of clinical trials, computer-assisted reading centers, and training. This work focuses on diabetes-related changes, although the techniques have broader applicability. Retinal features, including the vasculature, vessel branching/crossover locations, optic disk, and fovea are extracted automatically. The images are registered to sub-pixel accuracy using a 12-dimensional mapping that accounts for the unknown retinal curvature and camera parameters. The images are corrected for non-uniform illumination using a robust homomorphic surface fitting algorithm. The changes in non-vascular regions are segmented using an algorithm that is robust to relevant artifacts such as dust particles in the optical path. They are classified into five clinically significant categories using a Bayesian algorithm constrained by Markov Random Fields. A flicker animation overlaid with change analysis results allows qualitative and quantitative assessment by the user. A multi-observer validation on 43 image pairs from 22 eyes involving non-proliferative and proliferative diabetic retinopathies, showed a 96.83 % change detection rate, a 3.17 % miss rate, and a 17.65 % false alarm rate. The performance in correctly classifying the changes was 97.39 %.

This paper presents a disease-oriented evaluation of two recent retinal image registration algorithms, one for aligning pairs of retinal images and one for simultaneously aligning all images in a set. Medical conditions studied include diabetic retinopathy, vein occlusion, and both dry and wet age-related macular degeneration. The multi-image alignment worked virtually flawlessly, missing only 2 of 855 images. Pairwise registration, the Dual-Bootstrap ICP algorithm, worked nearly as well, successfully aligning 99.5% of the image pairs having a su#cient set of common features and 78.5% overall. Images of retinas having an edema and pairs of images taken before and after laser treatment proved the most di#cult to register.

This paper presents a collection of techniques, and lessons learned in the context of efficient migration of a large and complex computer vision code base developed off-line into an equivalent real-time implementation under a standard open-source operating system (Linux). Using creative linking strategies, it is possible to create a robust environment based on loadable kernel modules that enables simultaneous realization of real-time and off-line frame rate computer vision systems from a single code base.