Boosted Deep Convolutional Neural Networks Training

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Training neural networks, particularly convolutional neural networks (CNNs), is a time-consuming and resource-intensive process due to the need for large datasets and complex architectures, which can lead to cumbersome models that are not optimally suited for all applications, resulting in inefficient processing and resource utilization.

Innovation Solution

The use of a boosting approach with an ensemble of weak learners to train neural networks, where an auxiliary network computes boosting weights and provides them to other networks, allowing for online computation and reducing complexity, training time, and resource requirements, while augmenting datasets to enhance training without overfitting.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If a complex neural network architecture is used to improve classification accuracy, then manufacturing precision is improved, but device complexity increases

Engineering Contradiction:
Improveclassification accuracyVSAvoidnetwork architecture complexity
Core Design Contradiction:
Manufacturing precisionVSDevice complexity

Solution Approach 1:

The patent divides the complex neural network into multiple stages of training with different architectures. Initially, a simpler network is trained to establish basic classification capabilities, then progressively more complex networks are trained and combined. This segmentation allows the system to achieve high accuracy without requiring all complexity to be present simultaneously, resolving the contradiction between accuracy and complexity.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent implements preliminary training actions using simpler networks before deploying complex architectures. By pre-training with reduced complexity and progressively adding complexity in subsequent training stages, the system achieves high classification accuracy while managing device complexity through staged introduction of complex elements rather than immediate full complexity.

Inventive Principle:
Principle #10Preliminary action

2Manufacturing precision

If extensive training with large datasets is performed to improve classification accuracy, then manufacturing precision is improved, but loss of time increases

Engineering Contradiction:
Improveclassification accuracyVSAvoidtraining time
Core Design Contradiction:
Manufacturing precisionVSLoss of time

Solution Approach 1:

The training process is segmented into multiple stages, each using different dataset sizes and network complexities. Instead of requiring all tens of millions of training samples from the outset, the system progresses through stages where earlier stages use smaller datasets with simpler networks, then combines these with additional training data in later stages. This segmentation dramatically reduces the time required to achieve high accuracy compared to traditional approaches that require full dataset training from the beginning.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

Preliminary training actions are performed with smaller datasets and simpler architectures to establish baseline classification capabilities quickly. These preliminary results are then combined with additional training data in subsequent stages, allowing the system to achieve high classification accuracy without requiring the full extent of time-consuming training with all training samples simultaneously.

Inventive Principle:
Principle #10Preliminary action

3Manufacturing precision

If a complex neural network is used to improve classification accuracy, then manufacturing precision is improved, but use of energy increases

Engineering Contradiction:
Improveclassification accuracyVSAvoidprocessing energy
Core Design Contradiction:
Manufacturing precisionVSUse of energy by moving object

Solution Approach 1:

The patent segments the classification task across multiple training stages with progressively increasing complexity. Rather than deploying a single highly complex network that requires substantial energy for processing, the system uses a combination of simpler networks trained in different stages, each requiring less energy. The segmented approach allows high classification accuracy to be achieved through coordinated simpler components rather than a single energy-intensive complex network.

Inventive Principle:
Principle #1Segmentation

Data Source

PatentUS9805305B2Boosted deep convolutional neural networks (CNNs)
Publication Date: 2017.10.31 YAHOO ASSETS LLC
  • US9805305B2 patent drawing
  • US9805305B2 patent drawing
  • US9805305B2 patent drawing

AI summary

Briefly, embodiments of methods and/or systems of training multiclass convolutional neural networks (CNNs) are disclosed. For one embodiment, as an example, an auxiliary CNN may be utilized to form an ensemble with the collection as a linear combination. The linear combination may be based, at least in part, on boost prediction error encountered during the training process.