Screen-Based Computer Control with Pointing and Keyboard Actions

The patent replaces the mechanical DOM-element based control system with a visual-screen-based control system. Instead of requiring programs to access and manipulate DOM elements through complex APIs, the system captures visual screenshots of the screen, processes them through neural networks, and generates control actions based on visual content recognition. This substitution enables more intuitive and flexible task execution while maintaining control capability.

The patent uses visual screenshots as copies of the screen state instead of requiring access to the underlying DOM structure. By capturing and processing visual representations of the screen, the system achieves high task execution accuracy through neural network analysis of visual content, while simplifying data collection since screenshots can be obtained through standard screen capture methods without needing to parse or access complex DOM elements.

The patent implements neural network processing selectively for specific task execution steps rather than continuously processing all screen changes. The system captures screenshots and processes them through neural networks only when needed for decision-making, rather than maintaining constant automated analysis. This partial application of neural network processing achieves high automation capability while reducing unnecessary computational resource consumption.