Database Indexing and Query Optimization for Massive Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data query methods, such as disk scanning and partial indexing, are inefficient for massive data analysis, leading to slow query speeds and high resource wastage, as they either read all data from the disk or rely on limited indexing, failing to utilize associated information between multiple indexes effectively.
Innovation Solution
A data storage and query method that stores data in a row and column structure, constructs comprehensive indexes for all columns, and determines a query mode based on statistical information to optimize data retrieval, allowing for efficient querying through index information and reducing unnecessary disk reads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If disk scanning method is used to query data, then all data can be read from disk, but query speed is very slow and efficiency is poor
Solution Approach 1:
The patent applies preliminary action by pre-building comprehensive indexes for all columns before query operations. When data is stored, the system immediately constructs index structures that map column values to row locations. During query execution, these pre-built indexes enable direct location lookup instead of scanning all data, dramatically improving query speed while maintaining data completeness.
2Productivity
If partial index method is used to speed up query, then query time is reduced, but acceleration effect is limited because only portion of columns are indexed
Solution Approach 1:
The patent implements universality by creating comprehensive indexes that cover all columns in the dataset, rather than limiting indexing to only frequently queried columns. This multi-column indexing strategy ensures that any query condition on any column can be efficiently evaluated, providing universal query acceleration across diverse query patterns and improving both query speed and adaptability.
Solution Approach 2:
The patent merges multiple index structures into a unified comprehensive indexing system. Instead of maintaining separate partial indexes for different columns, the system combines index information from all columns into an integrated structure that can be leveraged by any query, maximizing the acceleration effect across all query operations.
3Productivity
If multiple indexes are built for multiple columns, then query optimization is possible, but associated information between multiple indexes is not utilized effectively
Solution Approach 1:
The patent merges multiple column indexes into a unified comprehensive index structure that inherently captures associations between columns. This unified structure allows the system to simultaneously evaluate multiple query conditions and leverage relationships between different columns during a single query operation, improving optimization effectiveness while simplifying index management compared to maintaining separate partial indexes.
Data Source
AI summary
Data storage and query method and device are disclosed, which facilitate a quick acquisition of query results through index queries at subsequent stages by establishing indexes for columns of a table. Furthermore, by scanning data in the table to obtain statistical information of data in the columns, this facilitates using the statistical information of the data in the columns to perform cost estimation in subsequent queries, in an attempt to obtain a data query mode that has the least cost and the best performance, thus improving query efficiency.