Decoupled Index Store for Cloud Data Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud storage services often impose limitations on the number, size, and configuration of indexes, making it difficult for users to create and manage indexes for data objects, which can limit query performance and user flexibility.
Innovation Solution
A data store indexing system that decouples the index store from the data store, allowing for separate optimization and configuration, with an index store that uses an in-memory cluster for fast queries and updates, and an indexing engine that applies updates based on configurable indexing rules, enabling automated refreshing and intuitive configuration via a user interface.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If cloud storage services maintain indexes for data objects, then query performance is improved, but the service imposes limits on the number, size, and usage of indexes which reduces user flexibility
Solution Approach 1:
The patent segments the index management functionality by separating the index store from the data store, allowing indexes to be maintained independently. This enables users to create and configure multiple indexes without being constrained by data store limitations, thus improving query performance while maintaining user flexibility.
Solution Approach 2:
The patent introduces an indexing engine as an intermediary component that automatically maintains indexes based on configurable rules. This intermediary layer shields users from the underlying storage service limitations, allowing flexible index configuration and automatic refresh operations without direct intervention in the data store, thereby resolving the contradiction between query performance and user flexibility.
2Ease of operation
If indexes are maintained in the data store, then query operations are facilitated, but the data store becomes more complex and harder to configure
Solution Approach 1:
The patent extracts the index store functionality from the data store, creating a separate indexing system. This extraction simplifies the data store configuration by removing index management responsibilities, while still facilitating easy querying through the dedicated indexing engine that operates independently on the separated index store.
Solution Approach 2:
The indexing engine serves as an intermediary that handles all index-related operations separately from the data store. This mediator manages index creation, configuration, and automatic refresh operations, making query operations easy while keeping the data store configuration simple and focused solely on data storage functions.
3Speed
If indexes are manually configured and refreshed, then query performance can be optimized, but the process requires significant manual intervention and maintenance effort
Solution Approach 1:
The patent implements self-service automation through the indexing engine, which automatically monitors data store changes and refreshes indexes based on configurable rules without manual intervention. This automated system maintains optimal query performance by continuously keeping indexes synchronized with the underlying data, eliminating the need for manual index management while preserving query optimization benefits.
Solution Approach 2:
The indexing engine employs feedback mechanisms by monitoring data store updates and automatically triggering index refresh operations when changes are detected. This closed-loop feedback system ensures indexes remain optimized for query performance while fully automating the refresh process, removing manual maintenance effort while maintaining high query speed.
Data Source
AI summary
Systems and methods are disclosed to implement an indexing engine that maintains an index in an index store for a storage object in a data store. In embodiments, the index store may be implemented using an in-memory storage cluster separate from the data store. The storage object may have multiple indexes, which may have different filtering or sorting criteria for the data. In embodiments, updates to the storage object are received as an update stream by the indexing engine. Based on configurable indexing rules, the indexing engine applies the updates to the appropriate indexes. To service a query to the data store, a query engine first retrieves a set of keys satisfying the query from the index store, and then data corresponding to the keys from the data store or another index. In embodiments, the index may be refreshed via touch updates of selected data in the storage object.