Sparse CNN Framework for Inking Applications
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional Neural Networks (CNNs) used in handwriting recognition and image classification face computational and memory burdens due to the high number of floating-point operations, despite generating sparse tensors, as conventional convolution processes do not adequately leverage sparsity.
Innovation Solution
A spatially sparse CNN framework that utilizes sparse tensor data structures and operations to reduce storage and computational costs by only processing non-zero elements, merging pooling and convolutional layers, and optimizing convolution operations for efficient processing of sparse input data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional convolution processes are used in CNNs, then the network can process input data, but the computational and memory storage burdens increase significantly due to the large number of floating point operations
Solution Approach 1:
The patent extracts and processes only the non-zero elements from the sparse input tensor, separating them from the zero elements. This is achieved by identifying the coordinates of non-zero elements and performing convolution operations only on these extracted elements, thereby eliminating unnecessary computations on zero-valued pixels that dominate in sparse inking data
Solution Approach 2:
Instead of performing complete convolution operations on the entire input tensor including all zero elements, the patent applies partial action by computing convolution only on the subset of non-zero elements. This partial processing significantly reduces the number of floating point operations while maintaining the essential information needed for handwriting recognition
2Reliability
If conventional convolution processes are used in CNNs, then the network can process input data, but the memory storage requirements increase due to the number of floating point mathematical operations
Solution Approach 1:
The patent extracts only the necessary non-zero elements and their coordinates from the sparse tensor, storing only this essential information rather than the complete dense tensor. This extraction approach significantly reduces memory storage requirements while preserving all information needed for accurate convolution operations
Solution Approach 2:
The patent segments the convolution process into multiple stages: (1) extracting non-zero element coordinates, (2) performing convolution on these coordinates, and (3) reconstructing the output. This segmentation allows memory to be used efficiently by processing smaller subsets of data at each stage rather than loading the entire dense tensor into memory
3Productivity
If sparse tensor processing is implemented, then computational and storage burdens are reduced, but the complexity of the convolution operation increases
Solution Approach 1:
The patent performs preliminary actions by pre-processing the sparse input tensor to identify and store the coordinates of non-zero elements before the main convolution operation. This preliminary step organizes the data in a way that simplifies subsequent convolution computations, as the algorithm only needs to process these pre-identified coordinates rather than searching through the entire tensor
Solution Approach 2:
The patent introduces an intermediary data structure that stores the coordinates and values of non-zero elements separately from the main tensor. This intermediary representation acts as a bridge between the sparse input format and the convolution operation, simplifying the computational process by providing direct access to only the relevant data elements
Data Source
Figure 1~3
Figure 4~5
Figure 6~7
AI summary
A spatially sparse convolutional neural network (CNN) framework is introduced to that leverages high sparsity of input data to significantly reduce the computational cost of applications that employ CNNs (e.g., inking applications and others) by avoiding unnecessary floating point mathematical operations. The framework, which is compatible with parallelized operations, includes (1) a data structure for sparse tensors that both (a) reduces storage burden and (b) speeds computations; (2) a set of sparse tensor operations that accelerate convolution computations; and (3) the merging of pooling and convolutional layers. Practical applications involving handwriting recognition and/or stroke analysis demonstrate a notable reduction in storage and computational burdens.