A new hybrid method is presented that combines the scale space filter (SSF) and Markov random field (MRF) for color image segmentation. The fundamental idea of the SSF is to use the convolution of Gaussian functions and image-histogram to generate a scale space image and then rind the proper interval bounded by the local extrema of the derivatives. The Gaussian function is with zero mean and varied standard deviation. Using the SSF the different scaled histogram is separated into intervals corresponding to peaks and valleys. The MRF makes use of the property that each pixel in an image has some relationship with other pixels. The basic construction of an MRF is a joint probability given the original data. The original data is the image that is obtained from the source and the result is called the label image. Because the MRF needs a number of segments before it converges to the global minimum, the SSF is exploited to do coarse segmentation (CS) and then MRF is used to do fine segmentation (FS) of the images. Basically, the former is histogram-based segmentation, whereas the latter is neighborhood-based segmentation. Finally, experimental results obtained from using SSF alone, MRF using iterated conditional mode (ICM), and MRF using Gibbs sampling are compared.