Multi-SoC Hand-Tracking Input Pipeline for Low-Latency AR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
AR systems lack effective user input modalities, making it difficult for users to indicate intent and invoke actions due to the absence of physical input devices like touchscreens or keyboards, limiting interaction capabilities.
Innovation Solution
Implement a hand-tracking input modality using computer vision-based Direct Manipulation of Virtual Objects (DMVO) and gestures, combined with a multi-SoC system where hand-tracking data is processed on a single SoC and shared across a cluster using IPC, minimizing latency and enabling synchronized input across multiple SoCs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If hand-tracking is implemented on multiple SoCs, then user interaction capabilities are enhanced, but processing latency increases
Solution Approach 1:
The system segments hand-tracking processing by designating a specific SoC (e.g., Qualcomm Snapdragon 845 or Samsung Exynos 9820) as the primary handler for computer vision and hand-tracking operations, while other SoCs in the cluster focus on application execution. This segmentation allows optimized processing location without requiring all SoCs to perform tracking, reducing overall latency while maintaining multi-SoC collaboration capabilities.
Solution Approach 2:
An intermediary communication mechanism is established between SoCs using the Android Shared Memory framework and IPC (Inter-Process Communication). The primary SoC processes hand-tracking data and shares results with other SoCs through this intermediary channel, eliminating the need for direct point-to-point communication between all SoCs and reducing communication overhead and latency.
2Reliability
If hand-tracking data is processed on all SoCs, then input synchronization is improved, but system complexity increases
Solution Approach 1:
Multiple SoCs are merged into a unified processing cluster where the Android Shared Memory framework provides a common memory space and IPC mechanism. This merging allows different SoCs to access hand-tracking data simultaneously without requiring complex inter-SoC coordination protocols, as the shared memory infrastructure handles synchronization automatically.
Solution Approach 2:
The system creates a universal hand-tracking processing model where any SoC can consume hand-tracking data through the standardized IPC interface, regardless of its specific role. This universality allows flexible deployment where different SoCs can be assigned different functions while maintaining synchronized access to tracking data through the common framework.
3Productivity
If computer vision processing is distributed across multiple SoCs, then processing capacity is increased, but data transmission overhead increases
Solution Approach 1:
The system extracts the computationally intensive hand-tracking and skeletal model inference operations from the distributed SoC cluster and consolidates them on a single designated SoC. Only the processed results (skeletal data, hand position information) are transmitted to other SoCs via IPC, rather than transmitting raw camera feeds or intermediate processing data, significantly reducing transmission overhead and energy consumption.
Data Source
AI summary
A multi-System on Chip (SoC) hand-tracking platform is provided. The multi-SoC hand-tracking platform includes a computer vision SoC and one or more application SoCs. The computer vision SoC hosts a hand-tracking input pipeline. The one or more application SoCs host one or more applications that are consumers of input event data generated by the hand-tracking input pipeline. The applications communicate with some components of the hand-tracking input pipeline using a shared-memory buffer and with some of the components of the hand-tracking input pipeline using Inter-Process Communication (IPC) method calls.