Neural Network Layer Culling with Bitmap-Guided Matrix Skipping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computing systems face challenges in processing large volumetric data sets efficiently, particularly in augmented and mixed reality applications, where high-frame rates and constrained memory and processing resources are required, leading to latency issues and motion sickness in users.
Innovation Solution
A sparse volumetric data structure is introduced, using a format like the sparse sexaquaternary tree, which tags voxels as occupied or empty, allowing for the removal of empty space and hardware acceleration, enabling faster processing and reduced latency in 3D rendering systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a separate GPU and computer vision subsystem are used in parallel, then processing capability is improved, but device complexity and memory resource consumption increase
Solution Approach 1:
The patent merges the computer vision subsystem and graphics processing into a unified system that shares memory resources and processing pathways. The depth buffer from the graphics pipeline is directly utilized by the computer vision algorithms, eliminating the need for separate memory spaces and reducing overall system complexity while maintaining parallel processing capabilities.
Solution Approach 2:
The graphics processing unit is designed to perform multiple functions - both traditional graphics rendering and computer vision processing. By making the GPU universal, the system eliminates the need for dedicated separate subsystems, reducing device complexity while preserving high processing capability for both graphics and vision tasks.
2Productivity
If high-frame rates are required for AR applications, then user experience is improved, but processing latency increases and causes motion sickness
Solution Approach 1:
The system performs depth buffer generation and voxelization operations in advance during the graphics rendering pipeline, so that when computer vision algorithms need this data, it is already prepared and available. This preliminary action reduces the critical path latency and enables high frame rates without causing motion sickness.
Solution Approach 2:
The patent establishes a continuous pipeline where graphics rendering and computer vision processing overlap in time. While graphics are being rendered, the depth buffer is being processed by vision algorithms, and results are being prepared for the next frame. This continuous useful action maintains high frame rates while minimizing latency.
3Manufacturing precision
If volumetric data structures are used to represent 3D space, then rendering accuracy is improved, but storage requirements and processing overhead increase
Solution Approach 1:
The patent implements a sparse voxel occupancy grid where only occupied volume elements are stored and processed, rather than representing the entire volumetric space densely. This porous approach maintains rendering accuracy for visible objects while dramatically reducing storage requirements by excluding empty space from the data structure.
Solution Approach 2:
The system applies different levels of volumetric representation to different regions of space based on their importance. High-density voxel representations are used only where objects actually exist, while empty regions use minimal or no representation. This local quality approach maintains rendering accuracy where needed while reducing overall storage requirements.
Data Source
AI summary
An output of a first one of a plurality of layers within a neural network is identified. A bitmap is determined from the output, the bitmap including a binary matrix. A particular subset of operations for a second one of the plurality of layers is determined to be skipped based on the bitmap. Operations are performed for the second layer other than the particular subset of operations, while the particular subset of operations are skipped.