The invention relates to a color image encryption system and an encryption and decryption method based on compression holography and belongs to the technical field of optical image encryption. The system comprises a laser device, a first reflecting mirror, a second reflecting mirror, a beam expanding and collimating system, a beam splitter, a first spatial light modulator, a second spatial light modulator, a third spatial light modulator, a fourth spatial light modulator, a first random phase mask, a second random phase mask, a third random phase mask, a high-speed image collecting device and a computer. Firstly, gray level images of three channels of color images are extracted and are arranged at the positions with different distances to a CCD to serve as object information needing to be encrypted, the three random phase masks are tightly attached to the three gray level images and then are used for modulating light beams containing the object information, the three light beams are interfered by a beam combining prism and reference light to generate a holographic image, and the gray level images of the three channels are simultaneously encrypted on the same holographic image. Lastly, decryption is carried out with a TwIST algorithm, and color image composition is carried out. According to the color image encryption system and the encryption and decryption method based on the compression holography, the color images can be encrypted through one-time exposure, and the encryption efficiency and the confidentiality degree of the encrypted information are improved.