Autonomous Storage-Side Indexing for JSON Document Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current document cloud systems face unsatisfactory performance for both OLTP and OLAP workloads, particularly when handling a mix of both, due to the need for manual indexing which is tedious, error-prone, and conflicts with default indexing strategies that either index everything or nothing, leading to performance degradation.
Innovation Solution
Implementing autonomous storage-side indices and optimization metadata within a data grid of storage cells that autonomously manage and tune their own indices, allowing for efficient filtering, projection, and joining operations, and providing a logical interface for application developers to access data without exposing the underlying physical implementation details.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If manual indexing is implemented to accelerate OLAP queries, then OLAP query performance is improved, but system complexity and operational overhead increase
Solution Approach 1:
The storage system autonomously creates, maintains, and optimizes indices without requiring manual DBA intervention. The system self-monitors query patterns, automatically generates appropriate index structures, and dynamically adjusts indexing strategies based on workload analysis, thereby resolving the contradiction between achieving fast OLAP queries and reducing operational complexity
Solution Approach 2:
The system dynamically changes indexing parameters such as index selection, data structure types, and maintenance frequency based on detected query patterns and workload characteristics. This adaptive parameter adjustment enables the system to optimize for OLAP performance when analytical queries are detected while avoiding unnecessary indexing overhead during OLTP-heavy periods
2Speed
If comprehensive indexing is applied to accelerate OLAP operations, then OLAP query speed is improved, but OLTP insert/update performance deteriorates
Solution Approach 1:
The system dynamically adjusts indexing strategies based on real-time workload detection. When OLTP operations dominate, the system minimizes or suspends index creation to preserve transaction throughput. When OLAP queries are detected, the system automatically creates and maintains indices to accelerate analytical operations, thereby resolving the performance trade-off between OLAP speed and OLTP productivity
Solution Approach 2:
The system applies different indexing qualities and strategies to different data subsets based on local access patterns. Frequently queried OLAP fields receive optimized indexing while fields primarily involved in OLTP operations use minimal or no indexing, allowing the system to achieve fast OLAP queries without uniformly degrading OLTP performance across all data
3Productivity
If no indexing is used to maximize OLTP performance, then OLTP transaction speed is improved, but OLAP query performance deteriorates
Solution Approach 1:
The storage system autonomously monitors query workloads and automatically creates indices when OLAP patterns are detected, eliminating the need for manual DBA intervention. This self-service capability ensures that OLTP performance is maintained through minimal default indexing while OLAP query speed is automatically improved when analytical workloads occur
Solution Approach 2:
The system performs preliminary analysis of query patterns and proactively creates index structures before OLAP queries suffer from performance degradation. By detecting workload shifts and pre-establishing appropriate indices, the system ensures fast OLAP response times without requiring comprehensive permanent indexing that would harm OLTP performance
4Measurement precision
If multiple indices are created to support diverse OLAP queries, then query selectivity is improved, but index maintenance overhead and storage space increase
Solution Approach 1:
The system dynamically changes indexing parameters including data structure selection (e.g., bitmap vs. B-tree), index granularity, and maintenance frequency based on query selectivity requirements and data characteristics. This enables the system to achieve high query selectivity for OLAP operations while optimizing storage consumption by avoiding unnecessary indices and using space-efficient structures for low-cardinality fields
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention relates to autonomous tuning of a data grid of documents in a database. Herein are techniques for storage cells to autonomously maintain local indices and other optimization metadata and algorithms to accelerate selective access into a distributed collection of documents. In an embodiment, each storage cell persists a respective subset of documents. Each storage cell stores, in memory, respective index(s) that map each item to location(s), in one or more documents of the respective subset of documents, where the item occurs. One or more computers execute, based on at least a subset of the indices of the storage cells, a data access request from a database management system. In an embodiment, a cloud of JSON document services provides an easy-to-use, fully autonomous JSON document database that horizontally and elastically scales to deliver fast execution of document transactions and queries without needing tuning by a database administrator.