Tokenized Cache System for Database Search Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search methods for large electronic data warehouses are inefficient, leading to slow search times and high resource usage, especially when dealing with millions to billions of records.
Innovation Solution
A tokenized cache system that parses textual items into tokens, stores these tokens in a memory cache with frequency values, and uses these values to select an optimal search token for limiting the search space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional search algorithms (e.g., Boyer-Moore) are used to search large databases, then search accuracy is maintained, but search time and resource consumption increase significantly
Solution Approach 1:
The patent segments the search process into two distinct phases: (1) a filtering phase using tokenized cache to quickly eliminate non-matching items, and (2) a verification phase using traditional search algorithms on the reduced subset. This segmentation allows the system to maintain search accuracy while dramatically reducing search time by limiting the application of computationally intensive algorithms to a small fraction of the total database.
Solution Approach 2:
The patent performs preliminary action by pre-tokenizing database items and storing their token representations in a cache before the actual search occurs. When a search query arrives, the system first converts the query into tokens and uses these to filter the database, creating a reduced search space before applying traditional search algorithms. This preliminary filtering action significantly reduces the workload of subsequent search operations.
2Reliability
If traditional search algorithms are applied to billions of records, then complete search coverage is achieved, but resource consumption becomes excessive
Solution Approach 1:
The patent divides the database into tokenized representations stored in a cache, separating the filtering function from the verification function. This allows the system to process billions of records by applying resource-intensive algorithms only to the small subset of items that pass the token-based filter, thereby maintaining search completeness while reducing resource consumption to manageable levels.
Solution Approach 2:
The tokenized cache serves as an intermediary layer between the search query and the full database. Instead of directly applying search algorithms to all records, the system uses token matches as an intermediate filtering step to identify candidate items, which are then verified against the original data. This intermediary mechanism preserves search reliability while minimizing resource usage.
3Reliability
If the entire catalog is searched for each query, then all matching items are found, but search speed decreases
Solution Approach 1:
The system performs preliminary tokenization of all catalog items and stores these tokens in a cache before searches are executed. When a query arrives, the tokens are used to quickly filter and identify candidate matching items before the full search is performed. This preliminary action enables the system to maintain match completeness while achieving significant speed improvements.
Solution Approach 2:
The patent segments the catalog into tokenized units and organizes them in a cache structure that enables rapid filtering. This segmentation allows the search process to be divided into a fast token-matching phase that identifies candidates, followed by a slower but necessary verification phase, thereby achieving both speed and reliability.
Data Source
AI summary
Methods of and systems for searching a catalog include parsing the items of the catalog into tokens, determining the frequency with which each token appears in the catalog, and storing the frequencies in a cache. Queries to the catalog are likewise parsed into tokens, and the tokens of the query string are compared to frequency values in the cache to identify a smaller search space within the catalog.