AI Model Preprocessing With Token Trees for Variable-Length Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based models face limitations in handling temporal sequences, require hard-coded position embeddings, are restricted by static computation graphs, and suffer from unexplainable biases due to the lack of negative example modeling in pre-training corpuses.
Innovation Solution
A preprocessing method tokenizes data into a tree structure, which is stored in a graph database, allowing dynamic programming and parallel processing, enabling AI models to handle variable-length inputs without static computation graphs and incorporating external knowledge for improved explainability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If Transformer-based models are used for language modeling, then high accuracy on language-based tasks is achieved, but the models require hard-coded position embeddings and cannot naturally handle temporal sequences
Solution Approach 1:
The patent replaces the mechanical Transformer architecture with a neural network architecture inspired by biological memory systems. Instead of using attention mechanisms and position embeddings, the invention uses a neural network with input layer, hidden layers, and output layer that processes sequences through learned temporal dynamics, substituting the mechanical Transformer structure with a biologically-inspired neural system that naturally handles temporal relationships
Solution Approach 2:
The invention changes the fundamental parameters of the language modeling approach by transitioning from static Transformer weight matrices to dynamic neural network parameters that adapt through learning. The system uses a vocabulary size parameter (e.g., 50,000 words) and trains the neural network to learn temporal patterns, replacing the fixed architectural parameters of Transformers with learnable parameters that capture temporal dynamics
2Adaptability or versatility
If Transformer models process large amounts of data, then the ability to understand and generate human language improves, but computational overhead increases due to hard-coded position embeddings
Solution Approach 1:
The patent eliminates the computationally expensive attention mechanisms and hard-coded position embeddings of Transformers by replacing them with a simpler neural network architecture. The system uses forward propagation through hidden layers with activation functions, which is computationally more efficient while maintaining the ability to process large datasets and learn language patterns
Solution Approach 2:
The invention extracts and removes the unnecessary computational components from the Transformer architecture, specifically eliminating the self-attention mechanisms and position embedding layers. The system keeps only the essential neural network components (input layer, hidden layers with activation functions, output layer) needed for language processing, reducing computational overhead while preserving language understanding capabilities
3Device complexity
If static computation graphs are used in AI models, then the model structure is simplified, but the models are restricted in handling variable-length inputs
Solution Approach 1:
The patent implements dynamic computation in the neural network by allowing the processing of variable-length sequences through the natural flow of data through hidden layers. The system can accept inputs of any length and process them through the neural network architecture, with the computation graph adapting dynamically to the input size rather than being constrained by a fixed static structure
Solution Approach 2:
The neural network architecture is designed to be universal in handling different input types and lengths. The same network structure (input layer, hidden layers, output layer) can process various sequence lengths and different language tasks, making the system multi-functional and adaptable without requiring separate architectures for different input scenarios
4Productivity
If pre-training corpuses contain only positive examples, then the model learns from available data, but unexplainable biases are introduced due to lack of negative example modeling
Solution Approach 1:
The patent introduces an intermediary mechanism in the form of a neural network with hidden layers that act as a mediator between input data and output predictions. This intermediate processing layer enables the system to learn from both positive and negative examples, providing explainable biases by showing how the hidden layers transform input features into predictions, rather than directly mapping inputs to outputs as in simpler models
Data Source
AI summary
Systems and methods relating to the application of AI related models to a corpus of data. The corpus of data is initially preprocessed by way of a tokenization process. This produces tokenized data that may then be grouped into groups of tokenized data. The tokenized data is then processed, either sequentially or in parallel, by one or more AI-related models. Each of the models implements a specific language task such as prediction, sentiment analysis, summarization, and others. All data adjustments, data processing, and data generation, both during the preprocessing and the AI model implementation, are stored such that other downstream processes can take advantage of the information generated by these processes.