Document Compression via Multi-Tiered Token Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current information retrieval systems face performance degradation due to the overhead of sophisticated query searching and scoring techniques, which require partial reconstruction of candidate documents to determine context, leading to inefficiencies in search performance.
Innovation Solution
A document compression system utilizing a multi-tiered mapping scheme that encodes documents into global and local token identifiers, allowing for compact storage and efficient decoding, thereby enhancing search performance without degrading query relevance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sophisticated query searching and scoring techniques are introduced to improve query relevance, then query relevance is improved, but search performance degrades due to additional processing overhead
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing document contexts, tokens, and metadata in the tokenspace repository during the document processing phase. This allows the query processor to retrieve pre-prepared information without performing complex reconstruction operations at query time, thus maintaining query relevance while improving search performance
Solution Approach 2:
The patent creates a compressed copy of document information in the tokenspace repository that contains essential contextual data, tokens, and metadata. This copy enables fast retrieval and scoring operations without requiring access to the original full documents, resolving the contradiction between maintaining query relevance and improving search performance
2Reliability
If partial reconstruction of candidate documents is performed to determine context of query terms, then query relevance is improved, but processing overhead increases leading to search performance degradation
Solution Approach 1:
The patent extracts only the necessary contextual information, tokens, and metadata from the original documents and stores them in the tokenspace repository. This extraction eliminates the need for partial document reconstruction during query processing, as the required context is already available in compressed form, thus improving search performance while maintaining query relevance
Solution Approach 2:
The tokenspace repository serves as an intermediary structure between the original documents and the query processor. It contains pre-computed contextual data that mediates the query processing operation, allowing the system to determine context of query terms without performing time-consuming partial reconstruction operations
Data Source
AI summary
The disclosed embodiments enable multi-stage query scoring, including “snippet” generation, through incremental document reconstruction facilitated by a multi-tiered mapping scheme. The mapping scheme includes a first mapping between unique tokens contained in a set of documents and unique global token identifiers (e.g., 32-bit integers) contained in a global-lexicon (i.e., dictionary). The mapping scheme also includes a second mapping between the global token identifiers and a set of fixed-length local token identifiers (e.g., 8-bit integers) contained in one or more mini-lexicons (i.e., sub-dictionaries). Each mini-lexicon is associated with a range of token positions in the tokenized documents. The first and second mappings are used to encode/decode documents into local token identifiers having fixed widths which can be compactly stored in the tokenspace repository. The use of fixed-length local token identifiers allows for fast and efficient decoding of tokenized documents.