The paper describes the motivation for, and the development of, a new way of comparing and assessing the accuracy of experts or algorithms at the task of medical image segmentation. Segmentation involves identifying key structures or objects in images, and is often carried out by experts who have considerable experience in the interpretation of images. Unfortunately, repeated segmentation by experts, and segmentations by different experts are usually different from each other, because of the difficulty of exactly delineating the structure of the image the same way each time.

“Medical imaging is one of our most powerful tools for gaining insight into normal and pathological processes that affect health.”

While many measures for comparing segmentations, such as volume, or spatial overlap, or distance between boundaries, have been developed, none have satisfactorily dealt with the variability of different expert segmentations and the lack of an intrinsic reference standard that reflects clinical imaging data.

The paper describes a principled algorithm for determining, from a collection of segmentations of the same image, a reference standard that reflects the segmentations and any prior information known about the anatomy under consideration and performance parameters by which the accuracy and precision of each segmentation generator may be assessed.

The paper describes a new methodology for assessing image segmentations. Until now, validation strategies have relied on synthetic phantoms that don’t reflect the variability of anatomy and pathology often encountered in clinical imaging, or else have used measures that don’t directly account for expert variability.

Since the paper was published, a number of groups, both nationally and internationally, have adopted the methodology and have applied it to the validation of image analysis. The manner in which the algorithm accounts for prior information about the anatomy under consideration, while simultaneously estimating a reference standard and performance parameters, has made it a useful tool that is easy to apply. I distribute source code to the algorithm to anyone interested in using it.

We have also found the algorithm to have wide applicability beyond the comparisons of segmentation, and have recently published an analysis of functional MRI activation maps using the technique.

Medical imaging is one of our most powerful tools for gaining insight into normal and pathological processes that affect health. Medical imaging as a field is starting to move beyond qualitative visualization, into quantitative assessment.

An emerging focus is the development of imaging biomarkers, using advanced image analysis algorithms in order to better predict outcomes of disease or treatment and intervention strategies.

A critical technological component is the creation and evaluation of segmentation algorithms that will enable high accuracy and precision image interpretation, with large volumes of data. This paper provides the validation strategy that will enable the fair and accurate comparison of different techniques, and rapid identification of the most promising methods.

I have been interested in medical image segmentation for more than 10 years. Over time, as the algorithms we developed for segmentation became more sophisticated and more accurate, incorporating ever-improving models of the anatomy and imaging physics, it became apparent that one of the factors preventing further rapid advance was the lack of a truly persuasive and convincing validation methodology.

I had tried all of the commonly used techniques and was unsatisfied with the state of the art. One common drawback was that existing methods were unable to directly account for differences in performance of experts. The initial aim for our work was to try to find a way to account for these differences, and eventually we succeeded in discovering the algorithm described in the paper.