Deep CNN Building Block Design with Branch and Skip Connections
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Convolutional Neural Networks (CNNs) face increased complexity in finding the right architecture and parameters due to a large search space, leading to longer training times and overfitting issues, with existing automated search techniques like reinforcement learning and evolutionary algorithms being computationally expensive.
Innovation Solution
A method and system that utilize a deep CNN building block with branch and skip connections, where undefined operations are treated as hyperparameters and automatically selected from a reduced set of possible operations, using random search to optimize and repeat the block to form a deep network, simplifying the search process and reducing the number of trials needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If automated search techniques like reinforcement learning or evolutionary algorithms are used to search through the architecture space, then the architecture discovery process is automated, but the computational expense increases significantly
Solution Approach 1:
The patent segments the architecture search problem into two independent parts: (1) searching for the optimal sequence of operations (architecture) and (2) optimizing the parameters for that architecture. This is achieved by first using random search to identify the best operation sequence, then separately optimizing parameters using gradient-based methods. This segmentation reduces computational expense by avoiding the need to simultaneously search both architecture and parameter spaces using expensive automated methods.
Solution Approach 2:
The patent employs random search as a cheap, computationally inexpensive method to explore the architecture space and identify promising operation sequences. Rather than using expensive reinforcement learning or evolutionary algorithms for the entire search process, random search provides a low-cost initial exploration that can be quickly executed to find viable architecture candidates before more expensive parameter optimization is performed.
2Reliability
If the search space includes all possible configurations of branch components, then the architecture search is comprehensive, but the training time increases
Solution Approach 1:
The patent divides the comprehensive search task into two phases: a fast random search phase that explores operation sequences, and a subsequent parameter optimization phase. This segmentation allows the system to maintain search completeness while reducing overall training time by handling architecture and parameter optimization separately rather than simultaneously exploring all configurations.
Solution Approach 2:
The patent performs preliminary random search to identify promising operation sequences before committing to full parameter optimization. This preliminary action filters the search space early, allowing the system to focus computational resources on optimizing parameters for already-viable architectures rather than exhaustively searching all possible configurations from scratch.
3Adaptability or versatility
If larger networks are used to accommodate more complex building blocks, then the architecture expressiveness increases, but the networks are more easily overfit to the data
Solution Approach 1:
The patent segments the model complexity into architecture selection (operation sequences) and parameter optimization. By separately handling these two aspects, the system can achieve high expressiveness through diverse operation sequences while controlling overfitting through dedicated parameter optimization with regularization techniques, preventing the need to increase network size unnecessarily.
Data Source
AI summary
A search framework for finding effective architectural building blocks for deep convolutional neural networks is disclosed. The search framework described herein utilizes a building block which incorporates branch and skip connections. At least some operations of the architecture of the building block are undefined and treated as hyperparameters which can be automatically selected and optimized for a particular task. The search framework uses random search over the reduced search space to generate a building block and repeats the building block multiple times to create a deep convolutional neural network.