Stream-Oriented Log Data Structure for Full-Text Search Inverted Index Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for indexing data streams and generating full-text search indexes are inefficient, consuming substantial resources and complicating data maintenance due to the need for replicating raw data and maintaining separate index and raw data files, leading to excessive storage and computational requirements.
Innovation Solution
A method that generates indexes for data streams in real-time, merging interval indexes into a single index and storing both indexes and data in a self-referential indexed stream file, reducing the need for multiple replicas and simplifying data maintenance by allowing deletion of both index and data in a single operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If raw stream data is replicated before indexing, then full-text search capability is achieved, but storage space and computational resources are substantially consumed
Solution Approach 1:
The patent merges the raw stream data and index into a single indexed stream data file. Instead of maintaining separate replicated raw data files and separate index files as in prior art, the invention combines them into one consolidated structure, eliminating the need for multiple replicas and reducing storage requirements while maintaining full-text search capability.
Solution Approach 2:
The indexed stream data file serves multiple functions simultaneously: it acts as both the raw data storage and the search index. This multi-functional structure eliminates the need for separate replication and indexing processes, reducing both storage consumption and computational overhead while preserving full-text search functionality.
2Adaptability or versatility
If separate index files and raw data files are maintained, then data can be indexed for search, but data maintenance complexity increases
Solution Approach 1:
By combining raw data and index into a single indexed stream data file, the patent eliminates the complexity of managing separate files. Deletion and maintenance operations can be performed on a single file rather than coordinating multiple replicas of both raw data and index files, significantly simplifying data maintenance while preserving search functionality.
3Reliability
If multiple replicas of raw data are indexed separately, then search coverage is comprehensive, but I/O operations and processing resources increase substantially
Solution Approach 1:
The patent merges multiple index structures into a single index within the indexed stream data file, eliminating the need to traverse and combine multiple separate index files during search operations. This reduces I/O operations and processing overhead while maintaining comprehensive search coverage across all stream data.
4Measurement precision
If interval indexes are created for every data record interval, then search precision is improved, but the number of indexes to traverse increases
Solution Approach 1:
The patent merges multiple interval indexes into a consolidated index structure within the indexed stream data file. This maintains the precision benefits of interval-based indexing while reducing the number of separate index traversals required, as the merged index provides a unified access path to all interval data.
Data Source
AI summary
Systems and methods are disclosed for efficiently indexing stream data to facilitate full-text search of the stream data. A stream comprises a plurality of intervals of log data records. An interval of log data records are indexed. The index and log data records for the interval are written to an indexed stream data file. The index for each interval contains pointers to the terms in the log data records for the interval. After a number of intervals of index and log data records have been written, a merge operation can merge the number of intervals of index into a single merged index. The merged index and intervals of log data records are written to the indexed data stream file. A full-text search index is generated by traversing and merging the interval indexes for the data stream.