Spreading Neural Network for Efficient Token Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network training methods face inefficiencies in processing large datasets and determining correlations between tokens, particularly in natural language understanding applications, where the size of the vocabulary exceeds what can be uniquely represented by orthogonal vectors.
Innovation Solution
The method involves processing an input matrix through a spreading component to produce a transformed matrix using a transformer neural network, followed by de-spreading to improve the neural network's output, allowing for the combination of correlated inputs without increasing the network size, and utilizing techniques like stacking and de-stacking to process multiple sequences in parallel.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional neural network training methods are used to process large datasets and determine correlations between tokens, then the network can handle natural language understanding tasks, but the processing efficiency is low and memory usage is high
Solution Approach 1:
The patent segments the neural network processing into distinct components: a spreading component that transforms input sequences into spread representations, a transformer neural network that processes correlations, and a de-spreading component that reconstructs outputs. This segmentation allows each component to be optimized independently, improving overall processing efficiency while managing memory usage through specialized function distribution across components.
2Adaptability or versatility
If the vocabulary size exceeds what can be uniquely represented by orthogonal vectors, then the neural network can handle larger vocabularies, but the representation becomes less precise
Solution Approach 1:
The patent transitions from traditional orthogonal vector representations to a spreading component that maps tokens into a transformed dimensional space. Instead of relying solely on orthogonal vectors in the original vocabulary dimension, the spreading component creates new representations that can accommodate larger vocabularies by distributing information across multiple dimensions, thereby maintaining representation precision even as vocabulary size increases.
3Measurement precision
If the neural network size is increased to improve correlation determination, then the accuracy of token correlation determination improves, but the network becomes more complex and resource-intensive
Solution Approach 1:
The patent introduces a spreading component as an intermediary between the input sequences and the transformer neural network. This intermediary transforms the input data into a format that is more amenable to correlation analysis, allowing the transformer network to achieve higher accuracy in determining token correlations without requiring a proportional increase in network size. The de-spreading component then acts as a second intermediary to reconstruct the final output, maintaining accuracy while controlling complexity.
Data Source
AI summary
Techniques for training neural networks are provided. According to one set of embodiments, a first array is processed in a spreading component to produce a second array, where a first dimension of the first array corresponds to at least one sequence of approximately orthogonal numeric vectors representing tokens, and where the spreading component combines values along the first dimension. The second array is processed in a transformer neural network to determine correlations between the sequence, which produces a third array. One or more batches of the third array are processed in a de-spreading component to produce a fourth array.