The invention discloses an automatic algorithm testing method and device, a computer system and a readable storage medium. Based on the technical field of testing, the method comprises the following steps: storing production data obtained from a production server into a data server, converting the production data into test data, deploying a test algorithm in a test server, and storing a test logicdata packet associated with a test demand; receiving target information output by the user side, determining a target test algorithm in the test server according to the target information, extractingtarget test data from the data server, and determining target test logic in the test server according to test requirements; and summarizing and packaging the target test logic and the target test data to form a test script, controlling a target test algorithm to run the test script to obtain a test result, calling a callback interface to generate a callback signal, and outputting the callback signal and the test result to the user side. According to the invention, the effect of automatically testing the algorithm is realized, and the problems of low working efficiency and high randomness of the current test are avoided.