Neural Network Layer Extension Without Changing Existing Outputs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for extending neural networks to perform additional computing tasks require substantial resource expenditure and can negatively impact the performance of the original task, whether through retraining or creating separate networks.
Innovation Solution
A system that electronically modifies an existing neural network by inserting additional layers that receive latent activations from the original layers, training these new layers to perform a different but related task without altering the original layers, allowing the network to perform both tasks simultaneously.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a completely separate neural network is created from scratch to perform another computing task, then the network can perform the new task, but substantial resources (computing time, processing power, training data) are required and the original network cannot perform the new task
Solution Approach 1:
The patent enables a single neural network to perform multiple computing tasks by adding a second set of layers that branch from the first set of layers. The first set of layers remains trained for the original task while the second set of layers is trained for the new task, allowing the network to serve multiple functions simultaneously without requiring separate networks for each task.
Solution Approach 2:
The neural network is divided into distinct functional segments: the first set of layers dedicated to the original computing task and the second set of layers dedicated to the new computing task. This segmentation allows independent training and optimization of each task-specific component while sharing the underlying feature extraction capabilities of the first set of layers.
2Adaptability or versatility
If the existing neural network is completely retrained to perform both computing tasks, then the network can perform both tasks, but the original task performance may be affected and substantial resources are required
Solution Approach 1:
The network architecture is segmented into a first set of layers that remains frozen and dedicated to the original task, and a second set of layers that is trained for the new task. This segmentation ensures that retraining for the new task does not affect the weights and parameters of the first set of layers, thereby maintaining stable performance on the original task while acquiring new capabilities.
Solution Approach 2:
The first set of layers is pre-trained for the original computing task before the network is extended. This preliminary training establishes a solid foundation of learned features that can be leveraged by the second set of layers for the new task, ensuring that the original task performance is preserved while enabling multi-functionality.
3Adaptability or versatility
If the existing neural network is completely retrained to perform both computing tasks, then the network can perform both tasks, but substantial resources (computing time, processing power, training data) are required
Solution Approach 1:
The first set of layers is pre-trained for the original computing task before the network is extended. This preliminary training establishes a solid foundation of learned features that can be leveraged by the second set of layers for the new task, significantly reducing the training time required compared to training from scratch or retraining the entire network.
Solution Approach 2:
Instead of retraining the entire neural network for the new task, only the second set of layers is trained. This partial training approach focuses computational resources only on the new task-specific components while leaving the first set of layers unchanged, thereby reducing overall training time and resource consumption.
4Adaptability or versatility
If a completely separate neural network is created from scratch, then the network can perform the new task, but device complexity increases and multiple networks must be maintained
Solution Approach 1:
The patent implements multi-functionality within a single neural network architecture by adding a second set of layers that branch from the first set of layers. This allows the network to perform both the original computing task and the new computing task through a unified architecture, reducing the need to create and maintain separate networks for each task.
Solution Approach 2:
The patent merges the functionality of multiple task-specific networks into a single unified network. The first set of layers handles the original task while the second set of layers handles the new task, both operating within the same network structure and sharing common computational resources, thereby reducing overall system complexity.
Data Source
AI summary
Systems and techniques that facilitate extension of existing neural networks without affecting existing outputs are provided. In various embodiments, a receiver component can access a neural network, wherein the neural network includes a first set of layers trained to perform a first computing task. In various instances, an extension component can insert a second set of layers into the neural network, wherein the second set of layers receive as input latent activations from the first set of layers. In various aspects, a training component can train, without changing the first set of layers, the second set of layers to perform a second computing task that is different from the first computing task.