A high-throughput system for segmenting nuclei using multiscale techniques

Automatic segmentation of cell nuclei is critical in several high-throughput cytometry applications whereas manual segmentation is laborious and irreproducible. One such emerging application is measuring the spatial organization (radial and relative distances) of fluorescence in situ hybridization (FISH) DNA sequences, where recent investigations strongly suggest a correlation between nonrandom arrangement of genes to carcinogenesis. Current automatic segmentation methods have varying performance in the presence of nonuniform illumination and clustering, and boundary accuracy is seldom assessed, which makes them suboptimal for this application. The authors propose a modular and model-based algorithm for extracting individual nuclei. It uses multiscale edge reconstruction for contrast stretching and edge enhancement as well as a multiscale entropy-based thresholding for handling nonuniform intensity variations. Nuclei are initially oversegmented and then merged based on area followed by automatic multistage classification into single nuclei and clustered nuclei. Estimation of input parameters and training of the classifiers is automatic. The algorithm was tested on 4,181 lymphoblast nuclei with varying degree of background nonuniformity and clustering. It extracted 3,515 individual nuclei and identified single nuclei and individual nuclei in clusters with 99.8 +/- 0.3% and 95.5 +/- 5.1% accuracy, respectively. Segmented boundaries of the individual nuclei were accurate when compared with manual segmentation with an average RMS deviation of 0.26 mu m (similar to 2 pixels). The proposed segmentation method is efficient, robust, and accurate for segmenting individual nuclei from fluorescence images containing clustered and isolated nuclei. The algorithm allows complete automation and facilitates reproducible and unbiased spatial analysis of DNA sequences. Published 2008 Wiley-Liss, Inc.