Neural Network Output Compression for Lower Feature Data Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks require significant bandwidth and memory for transferring and storing feature amount data, which is computationally intensive and inefficient.
Innovation Solution
An information processing apparatus with a computing unit and a compressing unit that irreversibly compresses feature amount data from at least part of the input, hidden, and output layers of a neural network, reducing data transfer and memory requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If feature amount data is transferred and stored in deep neural networks, then inference processing can be performed, but bandwidth becomes tight and large memory is required
Solution Approach 1:
The patent extracts and removes redundant information from feature amount data through irreversible compression. The compressing unit identifies and eliminates redundant features while retaining essential information needed for inference processing, thereby reducing data amount without completely sacrificing inference capability.
Solution Approach 2:
The patent changes the parameter representation of feature amount data by transforming high-dimensional feature vectors into compressed low-dimensional representations. This parameter transformation reduces the quantity of data while maintaining the essential characteristics needed for inference through learned compression mappings.
2Quantity of substance
If feature amount data is reduced through compression, then bandwidth and memory requirements decrease, but information loss occurs due to irreversible compression
Solution Approach 1:
The patent converts the potentially harmful information loss from irreversible compression into a beneficial feature selection process. By deliberately removing redundant and less important features through compression, the system achieves efficient representation that focuses on essential information, turning information loss into improved data efficiency.
Solution Approach 2:
The patent applies partial compression to only the most redundant portions of feature amount data rather than compressing all data uniformly. The compressing unit selectively compresses features based on their importance and redundancy, maintaining high fidelity for critical features while aggressively compressing less important ones.
3Adaptability or versatility
If deep networks with multiple hidden layers are constructed, then feature extraction capability improves, but data amount between layers becomes enormous
Solution Approach 1:
The patent implements nested compression where compression operations are embedded within the deep network architecture. The compressing unit is nested between hidden layers, progressively compressing feature representations at each stage while maintaining the hierarchical feature extraction capability of deep networks.
Solution Approach 2:
The patent segments the deep network into compression stages, where the compressing unit divides the feature processing into original feature extraction and compressed feature representation phases. This segmentation allows the network to maintain rich feature representations when needed while using compressed representations for data transfer between layers.
Data Source
AI summary
According to an embodiment, an information processing apparatus includes a computing unit and a compressing unit. The computing unit is configured to execute computation of an input layer, a hidden layer, and an output layer of a neural network. The compressing unit is configured to irreversibly compress output data of at least a part of the input layer, the hidden layer, and the output layer and output the compressed data.