The invention discloses an APP
automatic testing method based on multi-
modal perception and an Agent
system, and relates to the field of image recognition or understanding, and the method comprises the following steps: a real-
time perception layer carries out the real-time screenshot of a screen interface of an APP; when the interface changes,
processing the screenshot in three paths: detecting a control bounding box; performing visual semantic coding on the interface elements; synchronously extracting the text content, and generating a function
label. And fusing the three paths of output results to construct a UI tree. When a task instruction is input, the dynamic reasoning layer generates an action sequence according to the task instruction and the UI tree through a nerve-symbol
collaboration mechanism, updates a dynamic document and stores the dynamic document in a real-time
knowledge base. And the execution optimization layer converts the action of the action sequence into an operation code exclusive to the platform, executes the operation code, automatically marks an abnormal path at the same time, and updates an avoidance strategy to a real-time
knowledge base. According to the method, real-time sensing, dynamic reasoning and
closed loop execution are realized through a three-level architecture, and the generalization ability, the exception coverage and the execution efficiency are remarkably improved.