Modulation and demodulation techniques are described for an optical PSK heterodyne transmission system operating at 560 Mbits/s and 1.2 Gbit/s. Performance limitations affecting the receiver sensitivity in a 1.2-Gbit/s DPSK system, such as laser phase noise, phase modulation depth, IF center frequency deviation, and local laser power are studied. High receiver sensitivities for PSK systems, such as -46.9 dBm for 1.2-Gbit/s DPSK, –52.5 dBm for 560-Mbit/s DPSK, and –51.6 dBm for 560-Mbit/s carrier recovery PSK using an external fiber cavity DFB-LD module and a balanced receiver were achieved. The applicability of the Mach-Zehnder modulator as a phase modulator for 1.2-Gbit/s DPSK is also demonstrated. A 1.2-Gbit/s DPSK transmission of over 100 km, using polarization diversity with novel polarization-insensitive automatic frequency control in an attempt to overcome signal fading caused by polarization fluctuation in the transmitting fiber, is also described. A receiver sensitivity of less than -42.8 dBm and varying within 1.4 dB for all states of polarization was achieved. A multichannel high-definition TV (HDTV) transmission experiment using a DPSK polarization-diversity tunable receiver is described. © 1990 IEEE