Search Apparatus Using Frequency-Based Document Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search systems face challenges in efficiently processing large-scale document sets due to memory constraints and reduced search speed, especially when comparing query-specific document distributions with entire document sets.
Innovation Solution
A search apparatus and method that divides a document set into regions, calculates word frequencies within these regions, and identifies characteristic words by narrowing down the data to be read during search, using an abstract matrix storage unit, region upper limit calculation unit, word frequency calculation unit, and document frequency reference unit.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all document words are stored in memory for high-speed search, then search speed is improved, but memory capacity requirements increase significantly
Solution Approach 1:
The patent divides the large document set into multiple regions based on document frequencies. The word document matrix is segmented into multiple sub-matrices, each representing a specific frequency range. This allows the system to process and store data in smaller, manageable chunks rather than requiring all data to be loaded into memory simultaneously, thus reducing memory capacity requirements while maintaining search efficiency.
Solution Approach 2:
The patent pre-calculates and stores frequency information in an abstract matrix before the actual search operation. By performing frequency calculations in advance and organizing data by frequency ranges, the system prepares the data structure optimally for quick access during search operations, eliminating the need to load and process entire document sets during searching.
2Measurement precision
If the entire document set is processed to ensure accuracy, then measurement precision is improved, but processing time increases
Solution Approach 1:
The patent segments the document set into frequency-based regions and processes only the relevant regions for each search query. By dividing the large document set into smaller frequency ranges and using region upper limit calculations, the system can determine which regions need to be examined, thereby reducing processing time while maintaining accurate frequency measurements for the queried keywords.
Solution Approach 2:
The patent implements a region upper limit calculation that determines the minimum necessary processing scope. Instead of processing the entire document set, the system calculates and processes only up to the upper limit of relevant frequency regions, achieving sufficient accuracy for search purposes without the excessive time cost of complete processing.
3Productivity
If document data is read sequentially to improve access speed, then data access efficiency is improved, but flexibility in processing different query types is reduced
Solution Approach 1:
The patent introduces a frequency-based dimensional organization to the data structure. By arranging document words in frequency-ordered regions rather than traditional sequential or alphabetical ordering, the system enables efficient sequential access within frequency ranges while also allowing flexible navigation across different frequency dimensions. This multi-dimensional organization supports both fast access and versatile query processing.
Data Source
AI summary
Provided is a search apparatus, a search method, and a program that can improve search speed for a document set even when an object to be searched is a large-scale document set. A search apparatus, in an embodiment, includes an abstract matrix storage unit, a word frequency calculation unit, and a document frequency reference unit.


