Neural Network Matrix Culling for Low-Latency 3D Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer systems face challenges in processing large 3D data sets for augmented, virtual, and mixed reality applications due to constraints in memory, processing resources, and the need for high-frame rates, leading to latency issues that can cause motion sickness and hinder the processing of volumetric data effectively.
Innovation Solution
A sparse volumetric data structure is introduced, utilizing a format like the sparse sexaquaternary tree, which tags voxels as occupied or empty, allowing for the removal of empty space and enabling hardware acceleration, reducing storage requirements and processing latency by using a unified computer vision and graphics pipeline.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a separate GPU and computer vision subsystem are used in parallel, then processing capability is improved, but device complexity and latency increase
Solution Approach 1:
The patent combines computer vision processing and graphics rendering into a single unified pipeline that shares memory resources and processing stages. This integration eliminates the need for separate GPU and computer vision subsystems, reducing system complexity while maintaining high processing capability through resource sharing and coordinated operation.
2Measurement precision
If dense volumetric data structures are used, then measurement precision is improved, but memory usage and processing latency increase
Solution Approach 1:
The patent segments volumetric data into discrete voxels that can be individually tagged and processed. This segmentation enables selective processing of only relevant volumetric elements, reducing overall processing latency while maintaining precision where needed. The voxel-based approach allows the system to handle large volumetric datasets efficiently by processing them in manageable units.
Solution Approach 2:
The patent applies different processing qualities to different regions of volumetric data by tagging voxels as occupied or empty. This local quality approach ensures high measurement precision for occupied voxels that contain relevant information, while using simplified representations for empty regions, thereby reducing memory usage and processing latency without sacrificing overall data accuracy.
3Reliability
If all matrix operations are performed, then calculation completeness is improved, but processing time and power consumption increase
Solution Approach 1:
The patent extracts and removes redundant matrix operations from the processing pipeline by using tagged volumetric data to identify which calculations are actually necessary. Empty or irrelevant voxels are excluded from matrix operations, eliminating unnecessary computations while maintaining calculation completeness for all occupied regions. This selective approach significantly reduces processing time and power consumption.
Solution Approach 2:
The patent performs partial matrix operations only on the necessary subset of volumetric data identified through tagging. Rather than computing all possible matrix operations on the entire volumetric dataset, the system performs calculations only on occupied voxels and their relevant neighbors, achieving sufficient calculation completeness with reduced processing time and energy expenditure.
4Speed
If high-frame rates are maintained, then user experience is improved, but power consumption and processing load increase
Solution Approach 1:
The patent implements periodic updates of volumetric data by tagging voxels and only reprocessing regions that have changed since the last frame. This periodic action approach allows the system to maintain high frame rates by skipping redundant processing of static regions, thereby reducing power consumption while preserving the appearance of continuous high-speed rendering.
Data Source
AI summary
An output of a first one of a plurality of layers within a neural network is identified. A bitmap is determined from the output, the bitmap including a binary matrix. A particular subset of operations for a second one of the plurality of layers is determined to be skipped based on the bitmap. Operations are performed for the second layer other than the particular subset of operations, while the particular subset of operations are skipped.