Multi-SoC Hand Tracking With Shared Memory for Low-Latency AR Input
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
AR systems lack effective user input modalities, making it difficult for users to indicate intent and invoke actions due to the absence of physical input devices like touchscreens or keyboards, limiting interaction capabilities.
Innovation Solution
Implement a hand-tracking input modality using computer vision-based Direct Manipulation of Virtual Objects (DMVO) and gestures, combined with a multi-SoC system where hand-tracking data is processed on a single SoC and shared across a cluster, minimizing latency through synchronized shared-memory buffers and IPC bridges.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If hand-tracking is implemented using computer vision, then user interaction capabilities are enhanced, but processing latency increases
Solution Approach 1:
The system divides the hand-tracking processing into separate functional components: a first SoC executes hand-tracking operations (camera capture, pose estimation, hand detection) while a second SoC handles application logic. This segmentation allows specialized optimization of the vision pipeline and reduces overall processing latency by distributing computational workload across dedicated hardware units.
Solution Approach 2:
A shared-memory buffer acts as an intermediary between the first SoC (hand-tracking) and second SoC (application) processors. This intermediary enables efficient data exchange with minimal copying overhead, reducing communication latency while maintaining synchronized access to hand-tracking data across multiple processing units.
2Power
If multiple SoCs are used for processing, then computational power is increased, but system complexity increases
Solution Approach 1:
The multi-SoC system is segmented into functionally specialized units: a first SoC dedicated to hand-tracking operations and a second SoC dedicated to application execution. Each SoC has dedicated hardware components (cameras, processors, memory) optimized for its specific function, which simplifies the overall system architecture compared to a single monolithic device while providing enhanced computational power.
Solution Approach 2:
The patent describes a universal head-worn display system that can operate in multiple modes (AR, VR, hybrid) and support various interaction modalities (hand-tracking, gestures, voice, physical buttons). This multi-functional capability is achieved through a standardized multi-SoC architecture that can be configured for different applications, reducing overall system complexity through reusability.
3Speed
If hand-tracking data is processed locally, then response speed is improved, but power consumption increases
Solution Approach 1:
The system segments processing tasks by executing computationally intensive hand-tracking operations (image capture, pose estimation, hand detection) on a dedicated first SoC with optimized vision hardware, while lighter application logic runs on a second SoC. This segmentation enables fast local processing of critical vision data while distributing power consumption across specialized hardware units rather than requiring a single high-power processor.
Data Source
AI summary
A multi-System on Chip (SoC) hand-tracking platform is provided. The multi-SoC hand-tracking platform includes a computer vision SoC and one or more application SoCs. The computer vision SoC hosts a hand-tracking input pipeline. The one or more application SoCs host one or more applications that are consumers of input event data generated by the hand-tracking input pipeline. The applications communicate with some components of the hand-tracking input pipeline using a shared-memory buffer and with some of the components of the hand-tracking input pipeline using Inter-Process Communication (IPC) method calls.