Assuming for the moment that reliable gold standards are available, how should registration errors be reported? The answer to this question depends on the context in which the method is to be used. For a user interested only in a small anatomical structure, a useful validation study might be the average error at that structure's location, expressed as a real-world distance (e.g., the mean error in registering the anterior commissure was 3.4 millimeters). The root mean square (RMS) error at this point is a commonly used alternative to mean error. Neither of these measures gives a sense of the spread of errors. In some contexts, it might be critical for the user that the errors never exceed 2.0 millimeters. Reporting the most extreme error at the structure of interest is helpful in this regard, but may be misleading since the extremes of a distribution are not very reproducible, especially when the number of observations is small. A plot of the distribution of errors at the point of interest would be more helpful, and a formal statistical test, the Kolmogorov-Smirnov test, can be used to compare distributions of errors to determine whether they are significantly different . Errors may not be homogeneously distributed in space (e.g., see Fig. 5
of the chapter in this volume entitled "Biological Underpinnings of Anatomic Consistency and Variability in the Human Brain" taken from , so the most informative report would be a complete three-dimensional plot of all errors showing the direction and magnitude.
If an anatomical point of interest cannot be delineated in advance, it is impractical to report three-dimensional error distributions for every point in the image. Indeed, it is not even
FIGURE 1 Schematic illustration of the arrangement of fiducial channels within a stereotactic head frame. Prior to the advent of frameless stereotaxy (see the chapter "Registration for Image-Guided Surgery"), such frames were used routinely prior to certain neurosurgical procedures to allow MRI, CT, and PET data to be mapped into the frame of reference of the operating room. The frame is attached directly to the bones of the skull through incisions, ensuring that it cannot move between imaging sessions. Special channels within the frame, represented by the "N" shaped lines, can be filled with a copper solution to make them easily recognized in MR images and with a solution of positron emitting radioisotope to make them visible with PET. The frame is precisely engineered so that the four horizontal channels are all parallel to one another and so that planes passing through them meet at right angles. The diagonal channels are also parallel to one another and the plane containing them is perpendicular to the planes defined by the two channels on the right and by the two channels on the left. These geometric and physical constraints make it possible to register images acquired with the frame in place extremely accurately. By ignoring or eliminating the frame information from the images, such data can be used as an excellent gold standard for validating other registration methods [9, 11]. Potential limitations include the fact that the brain may move with respect to the skull  and the fact that the MR and PET imaging modalities may introduce nonlinear distortions (see the chapters "Physical Basis of Spatial Distortions in Magnetic Resonance Images" and "Physical and Biological Bases of Spatial Distortions in Positron Emission Tomography Images").
practical to show the distribution of three-dimensional errors for every single point. However, it is possible to show the mean or RMS errors for every point by using color coding. An example of this is shown in Fig. 15 of the chapter "Warping Strategies for Intersubject Registration." To convey the distribution of errors, separate color-coded maps for various percentiles of the distribution could be used. A more common strategy is to collapse all of the regional three-dimensional errors as if they had been measured at the same point. This allows the distribution of errors to be shown in a single plot. However, because of the nature of the registration problem, it is rare for errors to be similar throughout the image. Errors near the center are generally likely to be smaller than those near the edge. Consequently, unless maximum errors are tolerably small at all locations, some consideration of regional variability is warranted. The Kolmogorov-Smirnov test just mentioned can be used as a guide to whether error distributions in different parts of the image can be appropriately pooled as representatives of the same underlying distribution.
For rigid-body registration, errors in the formal parameters of the spatial transformation model (e.g., errors in pitch or errors in x-axis translation) are commonly reported. Since the error associated with any rigid-body transformation can be expressed as another rigid-body transformation, this is potentially a powerful way to summarize errors. However, several cautions are required. First of all, the decomposition of a rigid-body movement into elementary rotations and translations is not unique (see the chapter entitled "Spatial Transformation Models"). Consequently, errors in rotation around the x-axis computed by one author may not be comparable to errors in rotation around the x-axis computed by another author. A second problem is that errors in rotation angles ignore the underlying geometry of rigid-body transformations. The true rotational error is a single rotation around an axis that may well be oriented obliquely to the coordinate axes. A third problem is that translations are intertwined with rotations. In the presence of a rotational error, an apparent translational error may actually partially compensate and bring the images back into closer registration.
One approach that will lessen some of these problems is to use instantaneous elementary rotations computed using the matrix logarithm function instead of traditional sequential rotations when computing errors. Instantaneous elementary rotations are introduced in the chapter "Spatial Transformation Models" and can be defined unambiguously. The magnitude of the true rotational error around an obliquely oriented axis can be computed as the square root of the sum of the squares of the three instantaneous elementary rotations. Alternatively, both the magnitude and the direction of the rotational error can be captured in a three-dimensional plot of all three instantaneous elementary rotations. This may help to identify tendencies of registration methods to identify rotations better in some directions than in others. The problem of correlations between rotations and translations may be mini mized by computing translational errors at the center of the object (which may be far from the origin of the coordinate system). In general, rotational errors are likely to have the least effect near the center of an object and to produce errors in opposite directions more peripherally in the object. A plot of the magnitude of rotation against the magnitude of translation may help to assess the success with which correlations between these two sets of parameters have been eliminated.
One final issue that needs to be addressed in a validation study is the capture range of the method. Some methods will do extremely well if the images are already approximately registered, but will often fail completely when the initial misregistration exceeds a certain value. It is generally not appropriate to include the errors associated with such failures when characterizing registration accuracy. Instead, the magnitude of initial misregistration that can be tolerated should be reported as the capture range. For most methods, the capture range is sufficiently large that minimal efforts are required to ensure that the correct value will routinely lie within the capture range. Methods that have a very small capture range may need to be preceded by another less accurate method that is more robust.
Was this article helpful?