Aggregation Index Structure for Big Data Query Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies, such as traditional relational databases, search engines, and big data processing systems, fail to efficiently handle exploratory ad hoc queries of big data due to poor aggregation query performance, high latency, and inability to quickly process new data, leading to inefficient result return times and inadequate support for changing query conditions.
Innovation Solution
An aggregation index structure that partitions streaming data by grouping and slicing, using index metadata, a sliced data list, and a detailed data store to improve query efficiency, allowing for quick aggregation and insertion of new data, with support for OLTP and OLAP scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional relational databases are used for big data aggregation queries, then data consistency and transaction reliability are maintained, but query performance deteriorates with hour-level result return times and memory overflow issues
Solution Approach 1:
The patent segments the index structure into multiple levels (first-level index structures and second-level index structures) and divides data into partitions. This segmentation allows the system to handle large-scale data by processing smaller partitions in parallel, improving query efficiency while maintaining data consistency through structured organization.
Solution Approach 2:
The patent introduces a hierarchical dimension to the index structure with multiple levels. The first-level index structures point to second-level index structures, creating a multi-dimensional organization that enables efficient navigation and aggregation queries across PB-level data without overwhelming memory resources.
2Speed
If search engines like ElasticSearch are used for detailed data indexing and querying, then insertion and query performance are improved, but aggregation query performance deteriorates with serious performance bottlenecks
Solution Approach 1:
The patent segments the index into first-level and second-level index structures, where each level serves different query needs. This segmentation enables the system to optimize for both detailed data querying (using appropriate index levels) and aggregation queries (using the hierarchical structure to efficiently summarize data across partitions).
Solution Approach 2:
The hierarchical index structure serves multiple functions: it supports detailed data querying, aggregation queries, and handles various data scales. The same index structure can be used for both OLTP and OLAP scenarios, providing universal functionality without requiring separate specialized systems.
3Quantity of substance
If big data processing systems like Spark and HBase are used for large-scale data storage and querying, then data capacity and aggregation query flexibility are improved, but query efficiency deteriorates due to lack of effective indexing mechanism
Solution Approach 1:
The patent pre-establishes hierarchical index structures (first-level and second-level index structures) that organize data into partitions before queries are executed. This preliminary organization enables efficient query execution by providing pre-computed aggregation paths and reducing the need for full data scans during query processing.
Solution Approach 2:
The index structure dynamically adapts to different query requirements by leveraging its hierarchical nature. The system can navigate through different levels of the index structure depending on the specific aggregation query needs, providing dynamic query optimization without requiring static pre-computation for all possible queries.
4Loss of time
If pre-calculated results are used in OLAP tools like Kylin and Druid, then query response time is improved, but adaptability to changing query conditions deteriorates as they cannot meet change requirements in exploratory scenarios
Solution Approach 1:
The hierarchical index structure provides dynamic query support by allowing navigation through different levels based on actual query requirements. When query conditions change, the system can adaptively traverse the index structure at appropriate levels, providing flexible response to exploratory queries without requiring pre-computation for all possible scenarios.
Solution Approach 2:
By segmenting the index into multiple levels and partitions, the system can selectively process only the relevant partitions for each query. This segmentation enables efficient handling of changing query conditions by focusing computation on specific data segments rather than requiring complete re-computation or having fixed pre-calculated results.
5Productivity
If batch processing is used in big data processing systems, then offline data processing capability is improved, but real-time data insertion and processing capability deteriorates
Solution Approach 1:
The system pre-establishes the hierarchical index structure and data partitions, enabling both batch processing and real-time insertion to leverage the same organized framework. New data can be inserted and immediately indexed into the appropriate partitions and index levels, providing real-time processing capability while maintaining batch processing efficiency for large-scale operations.
Data Source
AI summary
Provided is an aggregation index structure and an aggregation index method for improving aggregation query efficiency. The aggregation index partitions streaming data through two dimensions of grouping and slicing, and then aggregates the partitioned data. The structure includes index metadata, a sliced data list and a detailed data store; the aggregation index method includes three parts: aggregation index definition, aggregation index creation and aggregation index query. The aggregation index structure and method provided by the present disclosure can greatly improve the efficiency of aggregation query, support the ad hoc aggregation query of PB-level data, complete the aggregation query of a large amount of data in seconds, support the insertion of new data at any time, and return the query results in minutes or even seconds latency in the event of changes of the query range conditions.