A BICM-ID system with 3-dimensional rotated BPSK constellation and signal space diversity (SSD) is proposed to combat the effect of Rayleigh fading. A new criterion based on mutual information is proposed to find the optimal rotation matrix, and the labeling that fits well with the outer code is presented. Simulation results show that at BER of 10-5 over a Rayleigh fading channel, with the code length of 192,000 bits and the iteration number of 100, the performance of the proposed system is only about 0.8 dB from the Gaussian-input Shannon limit and exceeds the limit constrained by the traditional QPSK input without rotation or SSD, at the spectrum efficiency of 1 bit/s/Hz.

A method to construct bit-interleaved coded modulation with iterative demapping and decoding (BICM-ID) is developed that approaches the Shannon limit very closely at high spectrum efficiency, where amplitude-phase shift keying (APSK) constellations are designed and chosen. For 1/2-rate 64APSK, the Es/N0 threshold derived through extrinsic information transfer (EXIT) charts is less than 0.55 dB away from the Shannon limit of the continuous-input additive white Gaussian noise (AWGN) channel, and exceeds the theoretical limit constrained by standard 64QAM-input.