Synchronization Stream Indexing for Near-Real-Time Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to maintain search indexes for near-real time data in high throughput environments, particularly in cloud computing scenarios, leading to inefficiencies in data synchronization and search performance.
Innovation Solution
A method is implemented where search requests are categorized into search granularity classes, and a data synchronization stream is indexed to generate a search index, allowing for near-real time data access and search performance through the use of a hash-code map to manage multiple search requests concurrently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data synchronization stream is indexed in real-time for near-real time search, then search speed is improved, but system complexity and resource consumption increase
Solution Approach 1:
The patent segments the search index into multiple granularity levels (fine-grained and coarse-grained indexes). The fine-grained index handles detailed search queries while the coarse-grained index handles broader search patterns. This segmentation allows the system to serve different search needs with appropriate index types, reducing the complexity of maintaining a single comprehensive real-time index while still providing fast search capabilities across different data granularities.
2Productivity
If multiple search requests are handled concurrently with high throughput, then productivity is improved, but data synchronization accuracy may deteriorate
Solution Approach 1:
The patent implements preliminary indexing actions where data is pre-processed and organized into multiple granularity indexes before search requests arrive. The system maintains pre-computed fine-grained and coarse-grained indexes that are updated incrementally as data arrives in the synchronization stream. This preliminary organization allows concurrent search requests to be handled efficiently without compromising data accuracy, as the indexing structure is already prepared and validated.
Solution Approach 2:
The patent introduces an intermediary indexing layer that mediates between the incoming data synchronization stream and multiple concurrent search requests. This intermediary maintains multiple granularity indexes that act as buffers, allowing search requests to be served from pre-processed data structures while the underlying data stream continues to synchronize. This intermediary layer isolates search operations from data synchronization operations, ensuring both high throughput and data accuracy.
3Measurement precision
If fine-grained search index is maintained for detailed search queries, then search precision is improved, but indexing time and resource usage increase
Solution Approach 1:
The patent segments the indexing workload into multiple granularity levels. Instead of maintaining a single fine-grained index for all search operations, the system creates fine-grained indexes for specific data subsets and coarse-grained indexes for broader queries. This segmentation allows the system to achieve high search precision for detailed queries when needed, while using coarser indexes for less demanding searches, thereby reducing overall indexing time and resource consumption.
Solution Approach 2:
The patent applies partial indexing by creating fine-grained indexes only for data portions that require detailed search capabilities, rather than indexing entire data sets at fine granularity. The system selectively maintains fine-grained indexes based on query patterns and data importance, using coarse-grained indexes for the remainder. This partial approach achieves sufficient search precision for critical queries while significantly reducing the time and resources required for comprehensive fine-grained indexing.
Data Source
AI summary
Embodiments of the present invention disclose a method, a computer system, and computer program products. A search request is received. One of one or more search granularity classes based on the subject matter of the search request is assigned. A data synchronization stream of data records to generate a search index based on the one or more search granularity classes is indexed. Responsive to indexing a predefined number of data records associated with the assigned search granularity class, a search based on the received search request is performed.