Touch Input Latency Reduction via Dynamic Delivery Timing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-touch enabled devices face high latency between touch events and display updates due to complex graphics processing pipelines, which is undesirable for user experience.
Innovation Solution
A method to process multi-touch input by determining the optimal time to deliver processed scan data to applications, animating graphical changes, and rendering them at the start of the next vertical blanking interval, using performance measurement and estimation techniques to maximize the likelihood of completing processing before the next frame deadline.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If multi-touch scan data is processed and delivered to applications for handling, then the display can be updated to reflect user touch events, but latency increases due to the time required for processing, handling, animating, and rendering before the next vertical blanking interval
Solution Approach 1:
The patent applies preliminary action by performing touch scan operations during the vertical blanking interval, before the display refresh begins. This allows touch data to be collected and prepared in advance, so that when the VBI ends and display updating starts, the touch processing can proceed immediately without waiting for the scan to complete, thereby reducing overall latency
Solution Approach 2:
The patent implements dynamics by dynamically adjusting the timing of touch data delivery to applications based on performance measurements. The system measures how long applications take to process touch data and uses this information to optimize when to deliver touch sets to applications, maximizing the likelihood of completing processing before the next frame deadline while adapting to varying application performance characteristics
2Adaptability or versatility
If multiple touch events occur simultaneously at different portions of the display, then multi-touch functionality is enabled, but it becomes difficult to split the display into different portions for independent software element processing
Solution Approach 1:
The patent applies segmentation by dividing the display into different portions and assigning different independent software elements to process interactions for each portion. This allows simultaneous touch events at different display locations to be handled by separate software components, reducing the processing burden on any single application and enabling parallel processing of multiple touch events
Solution Approach 2:
The patent implements universality by creating a universal touch processing framework that can handle multiple simultaneous touch events across the entire display. The system provides a unified interface that works consistently whether one finger or multiple fingers are touching the screen, allowing the same software infrastructure to serve multiple functions and touch configurations without requiring separate processing paths
3Reliability
If applications process large amounts of near-simultaneous incoming touch data, then accurate touch interaction handling is achieved, but the cost and complexity of software running on the device increases significantly
Solution Approach 1:
The patent applies preliminary action by pre-processing touch scan data to extract and organize time and location information before delivering it to applications. This preliminary processing reduces the amount of raw data that applications must handle, allowing them to focus on interpretation and response generation, thereby maintaining accuracy while reducing overall system complexity
Solution Approach 2:
The patent implements an intermediary layer between the touch scan hardware and applications. This intermediary processes and structures the incoming touch data, providing a standardized interface that applications can use consistently. The intermediary handles the complexity of dealing with multiple simultaneous touches and provides simplified data structures to applications, reducing their processing burden while maintaining reliable touch interaction handling
Data Source
AI summary
Systems, methods, and computer readable media to reduce latency in a graphics processing pipeline of an electronic device having a touch-sensitive input device and a display are described herein. In one embodiment, the method includes obtaining a temporal sequence of multi-touch scan data, processing the multi-touch scan data, determining an “optimal” time to deliver the processed multi-touch scan data to the relevant application(s) running on the device for processing, animating the resultant graphical changes that are determined by the application(s) based on the multi-touch input, and rendering the determined graphical changes to a frame buffer. In some embodiments, determining the “optimal” time for delivering the processed multi-touch scan data to the relevant application(s) comprises measuring the performance of the relevant application(s) in order to determine how likely it is that the necessary processing of the received multi-touch information may be completed before the conclusion of the next vertical blanking interval.