Module for configuring initial data is connected to the data storage module. Module for executing test cases is connected to the data storage module, message I / O module and protocol emulation module. Being connected to message I / O module, the message storage module obtains feedback message. Being connected to each module, status execution and control module carries out coordination. Connected to the message storage module, the message process module compares reference feedback message with parsed feedback message and outputs the compared result. Based on the test device, the invention also discloses method including editing test cases, configuring network parameters, obtaining reference feedback message, receiving feedback message from tested service system, comparing the said two pieces of message, and outputting the compared result. Features are: automatic testing, good expansibility of modules, and suitable to different platform.