Augmented Residual Blocks for Low-Complexity Deep Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks, such as residual networks (ResNets), face challenges in training efficiency as the number of layers increases, leading to higher latency.
Innovation Solution
The proposed solution involves a machine learning model that includes a plurality of residual blocks augmented with augmented weight blocks that sample intermediate features. These intermediate features are then applied to the augmented weight blocks to form intermediate outputs, which are combined to generate the final output.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of layers in a deep neural network is increased, then the network can capture more complex patterns and improve classification accuracy, but the training latency increases and computational complexity grows
Solution Approach 1:
The patent segments the deep neural network into residual blocks with skip connections, allowing the network to process information in manageable segments while maintaining overall depth for accurate pattern recognition
Solution Approach 2:
The patent introduces augmented weight blocks as intermediary components that sample intermediate features from residual blocks and combine them with original inputs, creating a mechanism that reduces training latency while preserving classification accuracy through feature augmentation
2Measurement precision
If the number of layers in a deep neural network is increased, then the network can capture more complex patterns and improve classification accuracy, but the model complexity and memory footprint increase
Solution Approach 1:
The patent divides the complex network into standardized residual blocks with identical structures, reducing model complexity through modular repetition while maintaining depth for accurate pattern recognition
Solution Approach 2:
The patent creates universal residual blocks that can be replicated throughout the network, where each block serves multiple functions: feature extraction, skip connection, and potential augmentation point, thereby reducing overall model complexity
3Measurement precision
If traditional deep neural networks are used, then classification accuracy can be achieved, but the memory footprint and computational resources required increase
Solution Approach 1:
The patent extracts and reuses intermediate features from residual blocks through the augmented weight blocks, eliminating redundant computations and reducing the effective memory footprint by leveraging previously computed features rather than recalculating them
Solution Approach 2:
The patent changes the parameter utilization strategy by introducing augmented weight blocks that dynamically sample and combine intermediate features, allowing the network to achieve accurate classification with more efficient parameter usage and reduced memory requirements
Data Source
AI summary
In some example embodiments, there may be provided a method that includes receiving, at a machine learning model, an input for a task of the machine learning model, wherein the machine learning model comprises a plurality of residual blocks augmented with a plurality of augmented weight blocks that sample intermediate features from the plurality of residual blocks; applying the input to the machine learning model to perform the task, wherein the applying comprises applying the plurality of intermediate features, which are obtained from the plurality of residual blocks, to the plurality of augmented weight blocks to form a plurality of intermediate outputs; and generating an output of the machine learning model, wherein the output is generated using at least on a combination of the plurality of intermediate outputs. Related systems, methods, and articles of manufacture are also disclosed.