The occurrence of phase separation in (100) ZnSe1-xSx and Zn1-yMgySe1-xSx layers grown by molecular beam epitaxy was found by transmission electron microscopy. The direction of the phase separation is [011], and the period of the composition modulation ranges from 300 to 500 angstrom. X-ray microanalysis of the two regions resulting from the phase separation showed one to be sulfur-rich and the other sulfur-deficient. The one-to-one correspondence of the wavy surface structure and the composition modulation suggests that the phase separation occurs via nonuniform incorporation of sulfur atoms into the wavy growth plane of the epilayer.