Neural Network Matrix Culling Using Bitmap-Based Operation Skipping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computer systems face challenges in processing and rendering high-frame rate, 3D data for augmented and mixed reality applications due to constraints in memory, processing resources, and power, leading to latency issues that can cause motion sickness and inefficiencies in handling large volumetric data sets.
Innovation Solution
A sparse volumetric data structure is introduced, using a format like sparse sexaquaternary trees, which tags voxels as occupied or empty, allowing for the removal of empty space and hardware acceleration, reducing storage needs and enabling faster processing and real-time updates in augmented, virtual, and mixed reality systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a separate GPU and computer vision subsystem are used in parallel, then processing capability is improved, but device complexity and power consumption increase
Solution Approach 1:
The patent combines the computer vision subsystem and graphics processing into a unified processor that can handle both volumetric data processing and rendering operations. This integration eliminates the need for separate parallel subsystems while maintaining processing capability for high-frame rate 3D data in AR/VR/MR applications.
Solution Approach 2:
The unified processor is designed to perform multiple functions including volumetric data processing, depth map generation, and graphics rendering within a single device. This multi-functional approach reduces system complexity while preserving the processing power needed for handling large volumetric datasets and real-time rendering.
2Productivity
If high-frame rate 3D data is processed and rendered, then visual quality is improved, but latency increases causing motion sickness
Solution Approach 1:
The system performs preliminary processing of volumetric data to generate depth maps and identify occupied voxels before the main rendering pipeline. This pre-processing allows the rendering stage to operate more efficiently with pre-computed depth information, reducing overall latency while maintaining high frame rates.
Solution Approach 2:
The patent extracts and processes only the essential depth information from volumetric data using sparse data structures that represent occupied space. By extracting only the necessary depth map data rather than processing complete volumetric datasets during rendering, the system achieves high frame rates with reduced computational latency.
3Loss of information
If complete volumetric data is stored and processed, then data completeness is improved, but memory requirements and processing time increase
Solution Approach 1:
The patent applies sparse volumetric data structures that store only occupied voxels rather than representing entire volumetric volumes. This local quality approach maintains data completeness for regions containing objects while using minimal memory for empty spaces, dramatically reducing memory requirements while preserving all necessary geometric information.
Solution Approach 2:
The system discards redundant representations of empty space in volumetric data while recovering and preserving only the essential occupied voxel information. This selective retention of data maintains complete geometric fidelity for objects while eliminating unnecessary memory consumption associated with representing empty volumetric regions.
Data Source
AI summary
An output of a first one of a plurality of layers within a neural network is identified. A bitmap is determined from the output, the bitmap including a binary matrix. A particular subset of operations for a second one of the plurality of layers is determined to be skipped based on the bitmap. Operations are performed for the second layer other than the particular subset of operations, while the particular subset of operations are skipped.