The newly proposed orthogonal time frequency space (OTFS) system exhibits excellent error performance on high-Doppler fading channels. However, the rectangular prototype window function (PWF) inherent in OTFS leads to high out-of-band emission (OOBE), which reduces the spectral efficiency in multi-user scenarios. To this end, this paper presents an OTFS system based on bi-orthogonal frequency division multiplexing (OTFS-BFDM) modulation. In OTFS-BFDM systems, PWFs with bi-orthogonal properties can be optimized to provide lower OOBE than OTFS, which is a special case with rectangular PWF. We further derive that the OTFS-BFDM system is sparsely-connected so that the low-complexity message passing (MP) decoding algorithm can be adopted. Moreover, the power spectral density, peak to average power ratio (PAPR) and bit error rate (BER) of the OTFS-BFDM system with different PWFs are compared. Simulation results show that: i) the use of BFDM modulation significantly inhibits the OOBE of OTFS system; ii) the better the frequency-domain localization of PWFs, the smaller the BER and PAPR of OTFS-BFDM system.