Object Storage Indexing for Search Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing full text search indexing systems face scalability issues, requiring increased hard disk drive space and the constant power supply of multiple machines, leading to high costs and inefficiencies as the index grows.
Innovation Solution
Implementing an indexing system that utilizes object storage and a distributed database, allowing indexes to be stored as objects with object keys, enabling efficient merging and distribution across multiple machines without the need for constant power, and leveraging encryption for security and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If hard disk drives are used to store index data, then the index can be searched and updated, but the system requires constant power supply and increases operational costs
Solution Approach 1:
The patent creates a copy of the index data in a different storage format (columnar storage) that can be searched without requiring the original hard disk drives to remain powered on. The index is exported to a searchable format that maintains query capability while eliminating the need for continuous power supply to the source storage system.
Solution Approach 2:
The patent extracts the index data from the hard disk drive storage system and places it in a separate searchable storage system. This separation allows the index to be searched independently without requiring the original storage system to remain powered on, thus reducing continuous power consumption while maintaining search functionality.
2Quantity of substance
If more hard disk drive space is allocated to grow the index, then more information can be stored and searched, but the cost and complexity of the system increases
Solution Approach 1:
The patent changes the storage parameters by converting traditional row-based storage to columnar storage format. This parameter change allows for more efficient storage of index data, increasing storage capacity while simplifying the system architecture. The columnar format enables better compression and more efficient use of storage resources without adding system complexity.
3Reliability
If traditional hard disk drive storage is used, then data can be stored and accessed, but the storage costs are high compared to object storage
Solution Approach 1:
The patent replaces the mechanical hard disk drive storage system with an object storage system using columnar data format. This substitution maintains data accessibility through efficient query mechanisms while dramatically reducing storage costs. The object storage system with columnar format provides comparable or better access performance at a fraction of the cost of traditional hard disk drives.
Data Source
AI summary
Aspects of the current patent document include systems and methods to perform search in an index system. In one embodiment, an index system may be implemented in an object storage. A distributed database index is used in conjunction with the object storage. In some cases, data stored in the distributed database may be encrypted and moved to object storage. The object storage stores a plurality of blocks containing words. Each block can contain a large number of words, such as one million words.