Neural Architecture Search for Dense Prediction Cells With Cached Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing architecture search techniques for dense image prediction tasks are computationally inefficient and ill-suited for high-resolution imagery due to large search spaces and reliance on low-resolution proxy tasks, making them unsuitable for achieving state-of-the-art performance.
Innovation Solution
The system limits the search space to identifying the best architecture for a dense prediction cell by using a smaller backbone during the search, pre-training the backbone, and caching feature maps generated by the pre-trained backbone to reduce computational resources and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If existing architecture search techniques are used for dense image prediction tasks, then the search can be performed using standard methods, but the computational cost becomes infeasible due to large search spaces and high-resolution imagery requirements
Solution Approach 1:
The network architecture is segmented into a backbone portion and a dense prediction cell portion. The search process is divided into two phases: first searching for the optimal dense prediction cell architecture, then combining it with a pre-trained backbone. This segmentation reduces the overall search space and computational requirements.
Solution Approach 2:
The backbone network is pre-trained on image classification tasks before being combined with the searched dense prediction cell. This preliminary training establishes feature extractors that can be reused, avoiding the need to train the entire network from scratch and reducing the computational burden of the architecture search.
2Measurement precision
If the search space includes all possible network architectures for dense prediction tasks, then the best architecture can be found, but the search becomes computationally infeasible
Solution Approach 1:
The architecture search is segmented to focus only on the dense prediction cell portion rather than the entire network. By fixing the backbone and only searching for the optimal dense prediction cell architecture, the search space is dramatically reduced while still enabling discovery of high-performance configurations.
Solution Approach 2:
The dense prediction cell is extracted as a modular component that can be independently searched and optimized. This extracted module can then be combined with different pre-trained backbones, allowing the search to focus on the critical prediction component without redundant exploration of backbone variations.
3Use of energy by moving object
If low-resolution proxy tasks are used for architecture search, then computational resources are reduced, but the search results are not representative of the final dense image prediction task
Solution Approach 1:
Instead of using low-resolution proxy tasks that don't accurately represent the target task, the system uses pre-computed feature maps from a pre-trained backbone as input to the dense prediction cell during search. This copying of feature representations allows the search to be performed at full resolution with accurate task representation while avoiding redundant computation.
4Measurement precision
If a full-sized backbone is used during architecture search, then the final architecture performance is optimized, but the search process consumes excessive resources and time
Solution Approach 1:
The backbone network is pre-trained on image classification tasks before being used in the architecture search. This preliminary training establishes robust feature extractors that can be reused during the search process, eliminating the need to train the backbone during search and significantly reducing search time while maintaining final architecture performance.
Solution Approach 2:
Pre-computed feature maps from the pre-trained backbone are copied and used as input to the dense prediction cell during architecture search. This copying approach allows the search to proceed without re-processing images through the backbone, reducing search time while preserving the quality of features for accurate performance evaluation.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining neural network architectures. One of the methods includes obtaining training data for a dense image prediction task; and determining an architecture for a neural network configured to perform the dense image prediction task, comprising: searching a space of candidate architectures to identify one or more best performing architectures using the training data, wherein each candidate architecture in the space of candidate architectures comprises (i) the same first neural network backbone that is configured to receive an input image and to process the input image to generate a plurality of feature maps and (ii) a different dense prediction cell configured to process the plurality of feature maps and to generate an output for the dense image prediction task; and determining the architecture for the neural network based on the best performing candidate architectures.