Automated Lean CNN Architecture Evolution via L1 Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manually designing deep neural networks is a tedious and inefficient process due to the vast space of hyperparameters, requiring both architectural engineering skills and domain expertise, and existing automated methods do not effectively leverage the potential of sub-networks within these networks for optimal performance.
Innovation Solution
A system and method that iteratively adds layers and prunes filters using a structured L1 pruning strategy to find and evolve lean, efficient sub-networks within deep neural networks, imposing constraints on model weights to remove inefficient weights and leverage the lottery ticket hypothesis, resulting in a lean model architecture suitable for low-power devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual design of deep neural networks is performed with trial-and-error approach, then architectural engineering skills and domain expertise can be applied, but the process becomes tedious and inefficient due to the vast space of hyperparameters
Solution Approach 1:
The system performs self-service by automatically searching for optimal sub-networks within the deep neural network using L1 pruning. The algorithm autonomously identifies and evolves efficient architectures without requiring manual trial-and-error by human experts, thereby dramatically improving design efficiency and reducing the time required for network architecture development.
2Productivity
If the entire deep neural network is used for classification tasks, then comprehensive feature extraction is achieved, but inefficient sub-networks cannot be identified and computational resources are wasted
Solution Approach 1:
The system extracts the efficient sub-network from the larger deep neural network by applying L1 pruning to identify and remove redundant or inefficient filters and connections. This extraction process isolates the most effective computational pathways, improving classification efficiency while reducing unnecessary computational resource consumption associated with the full network.
3Device complexity
If L1 pruning is applied to remove filters with low L1 norms, then the number of parameters is reduced and lean architecture is achieved, but the network structure becomes more complex to optimize
Solution Approach 1:
The system performs preliminary action by applying L1 pruning during the network evolution process to preemptively remove inefficient filters before they can significantly impact performance. This early pruning action simplifies the network architecture progressively while maintaining optimization efficiency, as the pruning guidance is integrated into the evolutionary search process rather than applied as a separate complex optimization step.
Data Source
AI summary
Disclosed herein is a system and method for evolving a deep neural network model by searching for hidden sub-networks within the model. The model is evolved by adding convolutional layers to the model, then pruning the model to remove redundant filters. The model is exposed to training samples of increasing complexity each time the model is evolved, until a desired level of performance is achieved, at which time, the model is exposed to all available training data.