Multi-tenant Cloud Indexing via Distributed Sharding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increased volume and complexity of data in cloud environments pose challenges for efficient indexing and searching of structured data, particularly in multi-tenant systems, where traditional methods struggle to maintain high availability and scalability while minimizing storage space and maximizing search speed.
Innovation Solution
An elastic scalable architecture is implemented, using a distributed database with sharding and Lucene indexing, allowing for dynamic reassignment of tenants to shards, separate management of search and indexing components, and a publish protocol for efficient data updating and indexing, ensuring high availability and scalability without downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional database indexing methods are used in cloud environments, then data storage is simple, but search speed decreases and scalability is limited
Solution Approach 1:
The patent divides the indexing system into separate components: Lucene indexing service for search optimization, distributed database for data storage, and sharding for data partitioning. This segmentation allows each component to specialize in its function, improving search speed while managing complexity through modular architecture.
Solution Approach 2:
The patent introduces an intermediary indexing layer between the distributed database and search queries. Lucene serves as an intermediary that pre-processes and structures data for rapid retrieval, separating the storage complexity from the search performance requirements.
2Quantity of substance
If data volume increases in cloud environments, then more data can be stored, but indexing and searching becomes less efficient
Solution Approach 1:
The patent implements sharding to segment data across multiple databases and nodes. By dividing large data volumes into manageable shards, the system maintains indexing efficiency even as total data quantity increases, allowing parallel processing and distributed indexing operations.
Solution Approach 2:
The patent employs dynamic sharding and elastic scalability where the indexing system can adapt its structure and resources based on data volume. New shards can be added dynamically, and the Lucene index can be rebuilt incrementally without complete system downtime, maintaining productivity during growth.
3Adaptability or versatility
If multi-tenant systems are implemented, then resource utilization improves, but data isolation and security complexity increases
Solution Approach 1:
The patent uses sharding to segment data by tenant, creating isolated data partitions while sharing underlying infrastructure. This allows multiple tenants to coexist on the same system with improved resource utilization, while data management complexity is reduced through automated shard routing and isolation policies.
4Reliability
If high availability is maintained during updates, then system reliability improves, but storage and processing overhead increases
Solution Approach 1:
The patent performs preliminary actions by building Lucene indexes incrementally from source data before they are needed for search. Indexing operations are prepared in advance and can be staged, allowing high availability during updates by pre-computing index structures without requiring complete data duplication.
Solution Approach 2:
The patent creates copied index structures from source data in the distributed database. Lucene indexes are copies optimized for search, separate from the original storage data. This copying approach maintains high availability during updates by allowing index rebuilding without affecting source data storage requirements.
Data Source
AI summary
In an example embodiment, a method for indexing data is provided. Primary data and auxiliary data is received from a first tenant in a computer network. A first index is then created for the primary data and the auxiliary data from the first tenant. The first index is stored in a first shard. Then primary data and auxiliary data are received from a second tenant in the computer network. A second index is created for the primary data and the auxiliary data from the second tenant. The second index is then stored in the first shard. The first shard is then stored in a first instance of a distributed database, the distributed database comprising a plurality of instances, each instance operating on a different logical or physical device.