Binocular Camera Touch Prediction on Large Displays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Interactive delays in large displays due to hardware and software processing delays negatively impact user experience, and existing methods for predicting user interactions are either costly or less accurate.
Innovation Solution
A method and system using a binocular camera to predict touch interaction positions on large displays by acquiring arm and facial/eye movement video frames, extracting key frames, marking coordinates, and training an ECO-Lite neural network to predict interaction positions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional hardware and software processing methods are used for touch interaction, then system stability is maintained, but interaction delay increases and user experience deteriorates
Solution Approach 1:
The system performs preliminary action by predicting the user's touch position before the actual touch occurs. Through real-time tracking of arm movement, facial expression, and eye movement, the system anticipates where the user will touch on the display, allowing the system to prepare and reduce processing delay when the actual touch happens.
Solution Approach 2:
The patent replaces traditional mechanical touch detection systems with a vision-based prediction system. Instead of relying solely on hardware touch sensors and software processing, the system uses computer vision to track user movements and predict touch positions, substituting the mechanical sensing approach with an optical prediction approach.
2Measurement precision
If deep learning methods are used to predict user behavior, then prediction accuracy improves, but calculation quantity and processing time increase
Solution Approach 1:
The system segments the prediction task into multiple independent components: arm movement tracking, facial expression recognition, and eye movement tracking. Each component is processed separately through dedicated neural network models, and the results are combined to determine the final predicted touch position. This segmentation allows parallel processing and reduces the computational burden of a single large model.
Solution Approach 2:
The system uses partial action by selecting only the most relevant features from each tracking component for the final prediction. Instead of processing all possible movement parameters, the system identifies and utilizes key features that have the strongest correlation with touch position, reducing calculation quantity while maintaining prediction accuracy.
3Measurement precision
If multiple tracking components are integrated for prediction, then prediction accuracy improves, but system complexity increases
Solution Approach 1:
The system merges multiple tracking components (arm movement, facial expression, eye movement) into a unified prediction framework. Each component is processed by its own neural network model, and the outputs are integrated through a fusion mechanism to produce the final touch position prediction. This merging allows the system to leverage complementary information from different body parts while maintaining modular architecture for manageable complexity.
Data Source
AI summary
Disclosed is a method and system for predicting a touch interaction position on a large display based on a binocular camera. The method includes: separately acquiring arm movement video frames of a user and facial and eye movement video frames of the user by a binocular camera; extracting a video clip of each tapping action from the arm movement video frames and the facial and eye movement video frames and obtaining a key frame by screening; marking the key frame of each tapping action with coordinates to indicate coordinates of a finger in a display screen; inputting the marked key frame to an efficient convolutional network for online video understanding (ECO)-Lite neural network for training to obtain a predictive network model; and inputting a video frame of a current operation to be predicted to the predictive network model and outputting a touch interaction position predicted for the current operation.