Tokenized Text Indexing for Verified Machine Learning Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI systems generate hallucinations due to insufficient training data and lack of real-time verification, leading to inaccurate outputs that are not grounded in the original input data.
Innovation Solution
A tokenized database is created with compressed text chunks, indexed by chunkIDs and tokenIDs, allowing for efficient retrieval and comparison of original text to verify AI responses, thereby reducing hallucinations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a tokenized database with compressed text chunks is created, then the efficiency of retrieving and verifying original text is improved, but the device complexity increases due to the indexing system with chunkIDs and tokenIDs
Solution Approach 1:
The patent segments the text database into chunks, where each chunk is assigned a unique chunkID. Within each chunk, words are further segmented into tokens with unique tokenIDs. This segmentation allows for efficient retrieval by dividing the large database into manageable units that can be queried independently, improving productivity while managing complexity through structured organization.
Solution Approach 2:
The patent introduces an intermediary indexing system that maps between chunkIDs, tokenIDs, and the actual text content. This intermediary layer (the index structure) mediates between the query system and the stored text, enabling efficient retrieval without requiring direct access to the entire database, thus improving productivity while the index structure manages the complexity.
2Ease of manufacture
If AI systems use aggregate properties from training data, then the model training process is simplified, but the veracity of output decreases due to hallucinations and loss of original information
Solution Approach 1:
The patent performs preliminary action by creating and maintaining an indexed tokenized database of original text before AI generation. This pre-prepared index allows the system to quickly retrieve and verify original information during the generation process, ensuring veracity while keeping the training process simple by using this pre-organized data structure.
Solution Approach 2:
The patent implements feedback mechanisms where the AI system can query the tokenized database to verify facts and retrieve original information during generation. This feedback loop allows the system to check its output against the original training data, improving veracity while the database structure maintains training simplicity by providing on-demand access to original information.
Data Source
AI summary
A database of tokenized data is provided. The tokenized database has been trained with a chunk of original text with words that have been compressed with tokens corresponding to the words. The text chunk is assigned a chunkID and at least some of the words are assigned a tokenID. The tokenized database can be filtered based on the tokenIDs for the one or more tokenized words from a search query. Each tokenID exposes a list of blocksIDs. A chunk of original text corresponds to each of the chunkIDs. The one or more sentences are compared to each sentence of the list of tokenized sentences to rank sentences.