Overlapping nuclei segmentation based on Bayesian networks and stepwise merging strategy

This paper presents a new approach to the segmentation of fluorescence in situ hybridization images. First, to segment the cell nuclei from the background, a threshold is estimated using a Gaussian mixture model and maximizing the likelihood function of the grey values for the cell images. After the nuclei segmentation, the overlapping and isolated nuclei are classified to facilitate a more accurate nuclei analysis. To do this, the morphological features of the nuclei, such their compactness, smoothness and moments, are extracted from training data to generate three probability distribution functions that are then applied to a Bayesian network as evidence. Following the nuclei classification, the overlapping nuclei are segmented into isolated nuclei using an intensity gradient transform and watershed algorithm. A new stepwise merging strategy is also proposed to merge fragments into a major nucleus. Experimental results using fluorescence in situ hybridization images confirm that the proposed system produced better segmentation results when compared to previous methods, because of the nuclei classification before separating the overlapping nuclei.