RZ signal transmission in an anomalous region with periodic dispersion compensation is examined by a straight-line experiment in terms of the compensation ratio, the signal power, and the pulse width. The optimum condition enables single-channel 20-Gbit/s RZ signal and two-WDM-channel 20-Gbit/s signals (40-Gbit/s in total) to be transmitted over 5,520 km and 2,160 km, respectively. Numerical simulations with the assistance of a basic theory enables analysis of the experimental results. It is shown that the balance between the waveform distortion and the remaining Gordon-Haus jitter determines the optimum conditions to achieve the longest transmission distance. Excess dispersion compensation results in waveform distortion, while insufficient compensation causes a greater amount of remaining jitter. Moreover, spectrum deformation during propagation is experimentally and numerically clarified to have a large effect on the transmission performance, especially for WDM transmission.