The invention discloses a system and a method for verifying register transfer level (RTL) hardware of a video algorithm. The system comprises a test video sequence library, a test vector generator, a golden C language model, an RTL hardware model to be verified and a file comparator, wherein the test video sequence library is used for storing a test sequence required by verifying the design of the RTL hardware of the video algorithm; the test vector generator is used for selecting the test sequence from the test video sequence library according to the functional coverage of the algorithm, generating a test vector and outputting the test vector to the golden C language model and the RTL hardware model to be verified; the golden C language model and the RTL hardware model to be verified are used for respectively generating output after receiving the test vector and outputting the respective output to the file comparator; and the file comparator is used for comparing whether the output of the golden C language model is consistent with the output of the RTL hardware model to be verified or not, indicating that the RTL hardware passes verification if the outputs are consistent, and indicating that the RTL hardware does not pass verification if the outputs are inconsistent. By the system and the method, the efficiency and correctness of verifying the design of the RTL hardware of the video algorithm are improved.