Neural Network Search Device Compressing Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network structure search methods face challenges with increased sample size leading to exponential search space growth, making training more difficult and time-consuming, and require large memory due to the need for gradient calculation of hyperparameters when dealing with many candidate operations.
Innovation Solution
The approach compiles multiple operations into a single operation, reducing the search space dimensions by treating similar operations as a single compressed operation, and uses a performance predictor to determine high-performance architectures, thereby reducing calculation and memory requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple operations are treated as separate candidate operations in the search space, then the search space covers more architectural possibilities, but the search space dimensionality increases exponentially making training more difficult and time-consuming
Solution Approach 1:
The patent merges multiple operations into a single compressed operation by grouping operations that have similar computational characteristics or can be represented through parameter variations. This reduces the search space dimensionality while preserving the essential architectural search capabilities, directly addressing the contradiction between search coverage and dimensionality complexity
2Reliability
If candidate operations are weighted by calculation cellhyperparameters and calculated by gradient method, then the architecture search can optimize performance, but the amount of memory required becomes large
Solution Approach 1:
The patent extracts and eliminates the need for gradient-based hyperparameter calculation by using a compressed operation representation that directly encodes operational characteristics. This removes the memory-intensive gradient computation step while retaining the ability to evaluate and optimize architecture performance through the compressed representation
Solution Approach 2:
The patent changes the parameter representation from detailed operational parameters requiring gradient optimization to a compressed parameter space that directly captures essential characteristics. This parameter transformation reduces memory requirements while maintaining performance optimization capability
3Device complexity
If the search space is converted to one-hot vector representation, then the architecture can be represented compactly, but the search space grows exponentially making training more difficult
Solution Approach 1:
The patent merges operations into compressed representations that reduce the number of dimensions required for one-hot vector encoding. By grouping operations with similar characteristics, the compressed representation maintains compactness while significantly reducing the exponential growth of search space dimensions, thereby improving training efficiency
Data Source
AI summary
The neural network structure search device that searches for a neural network architectures includes a calculation compression unit that compiles multiple operations as candidates for search included in an operation space into a single operation, and the architecture determination unit that determines the architecture with high performance from the candidate architectures that include the compiled operations.