Remote Input System for Mobile Devices Using Screen Snapshot Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing remote control technologies for mobile devices do not allow for remote input of Chinese characters or keyboard input, limiting the efficiency and convenience of information input, especially in scenarios like real machine testing where multiple devices with different models need to be tested.
Innovation Solution
A remote information input method and system that synchronizes the screen of a remote input device with a terminal device, uses a server to determine the existence and coordinates of target keys in a screen snapshot, and sends click instructions to simulate keyboard inputs, enabling keyboard input on remote devices, including Chinese characters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a virtual keyboard is displayed on a mobile device and operated through a mouse, then English characters or numbers can be input remotely, but Chinese characters cannot be input and keyboard input functionality is limited
Solution Approach 1:
The patent captures a snapshot of the terminal device's screen showing the virtual keyboard layout, and uses image recognition to identify key positions and characters. This copying approach allows the remote input device to replicate the terminal's keyboard interface, enabling support for various input methods including Chinese characters without requiring the terminal to display a specific keyboard layout
Solution Approach 2:
The patent introduces an image recognition module as an intermediary between the remote input device and the terminal device. This module processes the keyboard snapshot image to extract key position information and character mappings, enabling the system to translate physical keyboard inputs into corresponding virtual keyboard actions on the terminal, thereby supporting multiple input methods including Chinese
2Reliability
If multiple terminal devices with different models need to be tested, then comprehensive testing coverage is achieved, but physical presence of all devices is required which reduces efficiency
Solution Approach 1:
The patent captures screenshots of the virtual keyboard interfaces from multiple terminal devices with different models and input methods, and uses image recognition to identify and map key positions. This allows a single remote input device to simulate interactions with multiple different terminal devices, achieving comprehensive testing coverage without requiring physical presence of all devices
Solution Approach 2:
The patent creates a universal mapping relationship between physical keyboard keys and virtual keyboard keys across different terminal devices by recognizing keyboard layout images. This universal mapping enables one remote input device to control and test multiple different terminal devices with various keyboard layouts and input methods, significantly improving testing efficiency
Data Source
AI summary
Remote information input method, device and system are disclosed. The method includes: receiving a screen snapshot of a terminal device and sending the screen snapshot to a remote input device; receiving a keyboard input instruction from the remote input device, the keyboard input instruction including an identification of the terminal device and a keyboard code; determining a corresponding target key in the terminal device based on the keyboard code and a key code mapping table; identifying the screen snapshot of the terminal device and determining whether the target key exists in the screen snapshot; acquiring a coordinate of the target key in a screen snapshot in response to determining that the target key exists in the screen snapshot; and sending a click instruction to the terminal device, according to the coordinate of the target key in the screen snapshot and the identification of the terminal device.