Indexed Data Sharding for Fast Multi-Tenant Content Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing indexing technologies for large data sets lack scalability, accuracy, and efficiency in content retrieval, leading to increased latency and reduced processing speed, especially in multi-tenant data storage scenarios.
Innovation Solution
The generation and management of data shards, which are manageable-sized representations of data sets reflective of a normal distribution, are created for tenant-specific scenarios, enabling efficient and scalable content retrieval processing, including training of an artificial intelligence classifier, with data shards being pre-loaded for real-time or near real-time searching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional indexing is used for large data sets, then content retrieval can be performed, but processing efficiency deteriorates and latency increases
Solution Approach 1:
The patent divides large data sets into smaller, manageable data shards that can be processed independently. Each shard is indexed separately, allowing parallel processing and reducing the computational burden on single indexing operations. This segmentation enables faster content retrieval by limiting the search space to relevant shards rather than scanning entire data sets.
Solution Approach 2:
The system performs preliminary indexing and shard creation before actual content retrieval operations. Data is pre-processed, segmented into shards, and indexed in advance, so that when retrieval is needed, the system can quickly query pre-indexed shards without performing heavy processing during the retrieval moment itself.
2Loss of time
If traditional indexing is used for large data sets, then content retrieval can be performed, but latency increases
Solution Approach 1:
By segmenting data into smaller shards, the system reduces the time required to scan and process data during retrieval operations. Each shard can be queried independently and in parallel, significantly reducing overall latency compared to searching a single large index.
Solution Approach 2:
The patent changes the structural parameters of the indexing system by introducing shard size thresholds, distribution strategies, and hierarchical organization. These parameter changes optimize the balance between index granularity and query performance, reducing latency while managing complexity through configurable parameters rather than fixed complex structures.
3Reliability
If indexing is performed for all content, then comprehensive search capability is achieved, but processing efficiency deteriorates
Solution Approach 1:
The patent segments the comprehensive index into multiple data shards distributed across different storage units. This allows the system to maintain complete indexing coverage for accurate search results while processing queries in parallel across multiple shards, thereby improving throughput without sacrificing search accuracy.
Solution Approach 2:
The system performs indexing on data shards selectively based on relevance criteria rather than uniformly indexing all data. For query processing, it retrieves and processes only the necessary shards that contain relevant information, performing partial indexing actions that maintain accuracy for needed results while improving overall processing throughput by avoiding unnecessary full-index operations.
Data Source
AI summary
The present disclosure relates to processing operations configured to uniquely utilize indexing of content to improve content retrieval processing, particularly when working with large data sets. The techniques described herein enables efficient content retrieval when working with large data sets such as those that may be associated with a plurality of tenants of a data storage application/service. Among other technical advantages, the present disclosure is applicable to train a classifier using relevant samples based on text search in tenant-specific scenarios, where accurate searching can be executed for content associated with one or more tenant accounts of an application/service concurrently in milliseconds even in instances where there may be millions of documents to be searched. As an example, exemplary data shards may be generated and managed for efficient and scalable content retrieval processing including training of a classifier (e.g., artificial intelligence classifier) and real-time (or near real-time) query processing.