A combination of optical frequency division multiplexing (FDM) and phase-shift-keying (PSK) homodyne detection can increase transmission capacity. However, phase sensitive transmission systems, especially repeatered ones, suffer from data-dependent optical amplitude fluctuation because it is converted to phase fluctuation by fiber nonlinearity. This paper discusses how this data-dependent amplitude fluctuation affects the error rate performance of optical FDM PSK homodyne detection systems. If only the optical amplitude fluctuation induced by phase modulators is taken into account, the allowable power fluctuation to keep the power penalty due to both cross-phase modulation only and a combination of cross-phase modulation and self-phase modulation at 0.5 dB at a bit error rate of 10(-10) is below 0.17 mW for BPSK homodyne detection and 0.09 mW for QPSK homodyne detection. QPSK homodyne detection is weaker in phase changing than BPSK. However, if only the amplitude fluctuation induced by the fiber chromatic dispersion is taken into account, the allowable number of repeaters to keep a 0.5-dB power penalty due to XPM at a bit error rate of 10(-10) is 1 for BPSK homodyne detection and below 5 for QPSK homodyne detection. QPSK homodyne detection is more durable than BPSK homodyne detection for the same bit rate per channel. Because the more symbol bit rate decreases, the more the optical amplitude fluctuation can be decreased.