Index Storage Across Heterogeneous Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing enterprise search systems face inefficiencies in data retrieval speed and storage capacity, as caching methods do not improve search speed for less frequently searched data, and multi-tiered indexing architectures vary retrieval speed based on expected search frequency, but do not optimize storage usage effectively.
Innovation Solution
Distributing data structures across heterogeneous storage devices based on characteristics such as search query operator frequency, access patterns, and complexity, allowing for the use of both high-performance and low-performance storage devices to provide search results, thereby optimizing storage and retrieval efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If caching is used to store frequently searched data in lower-latency storage devices, then search speed for frequently accessed data is improved, but total storage capacity remains limited by the higher-latency storage device
Solution Approach 1:
The patent segments the index data into multiple partitions that can be distributed across different storage devices. Each partition can be independently managed and stored on appropriate storage media based on access patterns, allowing the system to overcome the storage capacity limitations of individual cache devices while maintaining fast access for frequently queried data segments.
Solution Approach 2:
The patent introduces a new dimension of index organization by creating inverted indexes that map terms to document identifiers. This dimensional transformation allows the system to efficiently retrieve documents through term-based lookups while distributing the index structure across heterogeneous storage devices, effectively expanding usable storage capacity without sacrificing search performance.
2Speed
If multi-tiered indexing architecture is used to store data on different storage devices based on search frequency, then retrieval speed is optimized for different data types, but storage usage is not effectively optimized
Solution Approach 1:
The patent applies local quality by organizing index partitions with specific characteristics (such as term frequency distributions and access patterns) on storage devices that are locally optimized for those characteristics. High-access-frequency partitions are placed on faster storage media, while less frequently accessed partitions reside on lower-cost storage, creating a nuanced storage hierarchy that better matches actual access patterns than traditional multi-tiered architectures.
3Speed
If high-performance storage devices are used to improve search speed, then data retrieval is faster, but system cost increases due to expensive storage hardware
Solution Approach 1:
The patent implements partial action by loading only the necessary index partitions into high-performance storage based on actual query workloads. Rather than storing entire indexes on expensive fast storage, the system dynamically determines which index segments require high-speed access and places only those on high-performance storage devices, reducing overall system cost while maintaining adequate search performance.
Data Source
AI summary
Indices or data structures used by an enterprise search system are stored across heterogenous storage devices. One or more characteristics associated with a data structure and one or more characteristics associated with a search query operator supported by the data structure are considered when determining which storage device should store each data structure.