Mobile Gesture Control for Desktop Apps via Camera Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
User interfaces designed for high-resolution pointers like trackpads or mice are cumbersome to navigate with mobile devices due to finger obstruction, screen dirt, and potential scratching, especially when remotely controlling desktop applications.
Innovation Solution
A system using a mobile device to receive video streams from cameras, analyze user gestures, and send corresponding pointing device commands to a remote desktop application, potentially utilizing two cameras for enhanced control, including eye or eyelid motions, and leveraging hardware for motion vector analysis and focus lock features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a user directly touches the mobile device screen to control desktop applications, then the control precision improves, but the screen becomes obstructed by fingers and susceptible to dirt and scratches
Solution Approach 1:
The patent introduces an intermediary control mechanism where gestures captured by cameras (either on the mobile device or external) are translated into pointing device commands. This mediator layer allows users to control the desktop interface without direct screen contact, maintaining precision through gesture recognition while eliminating finger obstruction and screen damage risks.
Solution Approach 2:
The patent replaces the mechanical direct-touch interaction with an optical-based gesture recognition system. By substituting the mechanical contact system with camera-based gesture detection, the system achieves the same control function without physical contact, thereby avoiding screen obstruction and damage.
2Measurement precision
If multiple cameras are used to capture gestures from different angles, then gesture recognition accuracy improves, but device complexity and processing requirements increase
Solution Approach 1:
The patent makes the camera system multi-functional by using the same cameras for both gesture recognition and potential other device functions. The system can selectively activate cameras based on control needs, and the captured video streams serve multiple purposes including gesture analysis and potential display feedback, reducing the need for dedicated specialized hardware.
Solution Approach 2:
The patent implements a flexible camera activation strategy where not all cameras need to be active simultaneously. The system can use a single camera for basic gesture recognition and activate additional cameras only when enhanced accuracy is needed, thereby balancing precision requirements with device complexity and power consumption.
Data Source
AI summary
One embodiment of the present invention provides a system for using a mobile device to remotely control a desktop application that was configured for use with a pointing device. During operation, the system receives at a mobile device, from a user, a connection request to connect to a desktop application executing on a remote device. In response to the connection request, the system establishes a remote control session between the mobile device and the remote device. Next, the system receives, at the mobile device, a video stream from a first camera coupled to the mobile device. The system then analyzes the video stream, at the mobile device, to identify a gesture made by the user. Next, the system identifies, at the mobile device, a pointing device command corresponding to the gesture. Finally, the system sends the pointing device command to the desktop application executing on the remote device.