Shard-Aware Near Real-Time Indexing via Urgent Queues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increased volume and complexity of data in cloud environments pose challenges for efficient indexing and searching, particularly in real-time scenarios, as traditional methods struggle to handle large datasets and provide fast search capabilities across multi-tenant infrastructures.
Innovation Solution
The implementation of a near real-time indexing system that assigns each shard a queue and a near real-time manager, allowing for urgent data indexing and search operations. This system utilizes Lucene indexes, distributed databases, and elastic sharding to ensure high availability, scalability, and flexibility, enabling simultaneous indexing and searching across multiple tenants.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional indexing methods are used in cloud environments, then data storage is simplified, but search speed and real-time indexing capability deteriorate due to large data volume and complexity
Solution Approach 1:
The patent divides the indexing system into multiple independent shards, each handling a portion of the data. Each shard has its own queue and near real-time manager, allowing parallel processing and improving search speed while distributing system complexity across multiple manageable units rather than one monolithic system.
Solution Approach 2:
The patent introduces a new dimension of operation by implementing near real-time indexing alongside traditional batch indexing. This creates a multi-layered indexing approach where urgent data can be indexed immediately while non-urgent data follows traditional processing, resolving the speed-complexity contradiction by operating in different time dimensions.
2Productivity
If traditional batch indexing is used, then system resource utilization is efficient, but real-time search capability deteriorates
Solution Approach 1:
The patent implements a dynamic indexing system where the indexing mode can switch between batch and near real-time based on data urgency. The system dynamically allocates resources to different queues, allowing high productivity for non-urgent data while minimizing indexing delay for urgent data through the urgent queue processed by near real-time managers.
Solution Approach 2:
The patent prepares indexing resources in advance by maintaining ready-to-process near real-time managers and urgent queues. When urgent data arrives, the indexing action can begin immediately without waiting for batch processing cycles, reducing indexing delay while maintaining overall system productivity through pre-positioned resources.
3Reliability
If data is indexed in near real-time for urgent searches, then search capability improves, but system complexity and resource requirements increase
Solution Approach 1:
The patent segments the system into independent shards with dedicated queues and managers, isolating complexity within each shard. This modular approach allows each shard to maintain near real-time indexing capability without requiring the entire system to become increasingly complex, as each shard manages only its own data portion independently.
Solution Approach 2:
Each shard is self-sufficient with its own queue and near real-time manager, allowing it to handle urgent indexing requests independently without coordinating with other shards. This self-service capability reduces overall system complexity by eliminating inter-shard coordination overhead while maintaining high search availability through distributed independence.
Data Source
AI summary
In an example embodiment, data to be indexed in a distributed file system is received via a near real time publish application program interface (API). A shard responsible for the data to be indexed is determined. Then a message is generated in a shard queue corresponding to the shard responsible for the data to be indexed, the message indicating that data needs to be urgently indexed, the detection of the message in the shard queue by a near real time manager corresponding to the shard responsible for the data to be indexed causing the near real time manager to cause the data to be indexed.