Tiered Metadata Indexing for Lower-Cost Cloud Data Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The large size of metadata indexes in cloud computing environments leads to high storage costs, as they often exceed the size of the underlying data, necessitating a need for systems and methods to organize and make data searchable while reducing storage costs.
Innovation Solution
A tiered indexing system is implemented, where data objects are categorized into tiers based on importance levels, with different indexing parameters for each tier, including hot, warm, and cold tiers, adjusting storage and metadata amounts accordingly to balance accessibility and cost.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If indexes of metadata are created to increase organization and searchability of data, then data searchability is improved, but storage costs increase due to the large size of indexes
Solution Approach 1:
The patent segments the index into multiple tiers (hot tier, warm tier, cold tier) based on data access frequency and importance. The hot tier contains frequently accessed metadata with full indexing for fast searchability, while the cold tier contains less frequently accessed metadata with reduced indexing or compression to minimize storage costs. This segmentation resolves the contradiction by providing high searchability only where needed while reducing storage overhead for less critical data.
Solution Approach 2:
The patent applies different indexing quality levels to different portions of the metadata based on their access patterns. High-importance metadata receives comprehensive indexing with detailed attributes for optimal searchability, while low-importance metadata receives minimal indexing or is stored in a compressed format. This local differentiation of quality resolves the contradiction by optimizing search performance for critical data while minimizing storage costs for non-critical data.
2Ease of operation
If comprehensive metadata indexing is applied to all data objects, then searchability is maximized, but the index size exceeds the underlying data size leading to high storage costs
Solution Approach 1:
The patent implements a dynamic indexing strategy where the indexing level for each metadata object is adjusted based on its access frequency and importance. The system continuously monitors access patterns and dynamically repositions metadata between tiers, adjusting the indexing depth accordingly. Frequently accessed metadata maintains comprehensive indexing, while infrequently accessed metadata has reduced indexing. This dynamic approach resolves the contradiction by adapting index size to actual search needs rather than applying static comprehensive indexing to all data.
Solution Approach 2:
The patent changes the indexing parameters (such as index depth, compression ratio, and attribute inclusion) based on the importance level and access patterns of different metadata objects. High-importance metadata uses parameters that maximize searchability with minimal compression, while low-importance metadata uses parameters that reduce storage size with acceptable search performance. This parameter adjustment resolves the contradiction by optimizing the balance between index size and searchability for each metadata object based on its specific characteristics.
Data Source
AI summary
A system for data indexing includes memory and at least one computing device in communication with the memory. The computing device can store data objects in the memory. The computing device can assign an importance level to, or determine an applicable policy for, each data object by at least analyzing each data object to determine an importance level for each data object, designate each data objects having a high importance level as a first-tier data object, and designating each data object having a low importance level as a second-tier data object. The computing device can store subject data for the data objects by at least storing a first amount of subject data for each first-tier data object and a second amount of subject data for each second-tier data object, where the second amount of subject data is less than the first amount of subject data.