The invention relates to a radio-frequency power amplifier linearization system based on a broadband modulator. The system comprises a receiving device, a channel and an emitting device. The emitting device at least comprises a test data generation device, a predistortion device, a modulation device and a power amplification device. The receiving device at least comprises a synchronous demodulation device, a data recovery device and a predistortion parameter calculation device. According to the predistortion parameter calculation device, a predistortion coefficient vector A is acquired by the optimized variable step size LMS algorithm, and the predistortion device is configured through the acquired coefficient. After an emitted signal matrix U passes the predistortion device, the predistortion device output vector Z is acquired by that Z = UA. According to the system, on the basis that the original LMS algorithm is not modified, the segment interval is refined and the step size is replaced, the step size factors of the LMS algorithm are corrected, the predistorted out-of-band rejection is increased by about 3dB and is closer to the optimal coefficient, and the nonlinear predistortion compensation effect is improved.