Ground truth files are and should be generated by physicians that are experts in the specific imaging modality and anatomy, e.g., CT interpretation and pancreatic disease. Despite subjectiveness and variability concerns, manual outlines by human experts are often the only way to establish ground truth or a form of ground truth to which image segmentation techniques can be compared to. If pathology information is available, as in the case of patients undergoing surgery or biopsy, then it can be used to provide stronger evidence on the true tumor size and possibly shape although the two sources of information, radiology and pathology, are not directly correlated. Unfortunately, in the case of pancreatic cancer, many of the tumors will not be resectable and will only undergo treatment; actually these are the patients one is most interested in, if one wants to determine treatment effects and tumor response over time. As a result, manual outlines by experts are the only option for ground truth generation on helical CT scans of the pancreas.
Figures 4.10(a) and 4.10(b) show two tumor outlines generated independently for the same CT slice and the same case with a malignant mass at the
head and tail of the pancreas respectively (original images shown in Figs. 4.6(a) and 4.7(a)). These outlines are part of the ground truth files generated for the CT slices and used for segmentation validation. Variations in the outlines as the ones seen in Fig. 4.10 are expected and inevitable between experts and could make segmentation validation a strenuous task. Often there is no right or wrong answer and it is our recommendation that both are considered in an evaluation process.
Measures can and should be taken to increase the accuracy of this information and at a minimum remove external sources of variability or error. These measures include the following:
(i) Establish optimum and standard viewing and outline conditions in terms of monitor display and calibration, ambient light, manual segmentation tool(s), and image manipulation options.
(ii) Use all individual manual outlines for evaluation as a possible way to account for expert variability. For example, use both outlines shown in Fig. 4.10 for segmentation validation. Alternative options are to determine the union or overlap of outlines or use a panel of experts to obtain a consensus on one outline per image.
(iii) Provide all available information to the expert before he/she generates truth file.
(iv) Have experts perform initial outlines independently to avoid bias (any joint outlines are done in addition to the originals).
(v) Review the expertise of the "experts" and their physical condition prior to the initiation of the process (number of cases read within a certain time frame, familiarity with computer tools, training, fatigue).
(vi) Establish standard criteria and conventions to be followed by all experts in their outlines.
Ground truth files are generated for all cases in the designed database but for a selected number of image series and slices to reduce physician effort. Specifically, in the cases where there is no high-resolution series (#3), ground truth files are generated for the 5 enhanced CT slices of Series #2 that contain the pancreas. In the cases where the high-resolution series (#3) is available, ground truth files are generated for both the 8 mm slices of the pancreas in Series #2 and every other slice in Series #3 (10 slices total); "ground truth" for the intermediate slices of Series #3 is obtained by interpolation. (Ground truth could be extrapolated from the 4 mm slices to the 8 mm slices but slice registration would be required prior to this process.)
Truth files are images of the same size as the original slice and include (a) the location of the pancreas and outline of its shape; (b) the location of the pancreatic tumor(s), masses, or cysts, and their shape outline(s) (Fig. 4.10); (c) the location and identification of neighboring organs and their shape outlines; (d) the identification of any vascular invasion and metastases sites and outlines. Truth files are generated using a computer mouse to outline the areas of interest on CT slices that are displayed on a high resolution (2048 x 2560 pixels) and high luminance computer monitor. Pixels in the ground truth files are assigned a specific value that corresponds to an outlined organ or structure, e.g., a gray value of 255 is assigned to the pixels that correspond to the outline of the pancreatic tumor(s), a gray value of 200 to the pixels that correspond to the outline of the normal pancreas, etc.
Was this article helpful?