The invention discloses a transform domain communication system and a realization method thereof. The transmitting end includes a first random phase spectrum sequence generation module and a first inverse Fourier transform module, and is characterized in that it also includes a first CCSK modulation module, a second CCSK modulation module, an imaginary multiplier module and a tributary signal superposition module; The receiving end includes a second random phase spectrum sequence generation module, a signal conjugate module, a Fourier transform module, a first multiplier module, a second inverse Fourier transform module, a real number data extraction module, an imaginary number data extraction module, a first a peak search module and a second peak search module. The system and method of the present invention utilizes the characteristic that the maximum value in the circular cyclic autocorrelation function is a real number, and loads the basic modulation waveform through the real number branch and the imaginary number branch respectively to load different data streams, thereby improving the data transmission rate of the transform domain communication system , thereby improving the system spectrum utilization.