Virtual Column Indexing for Faster Distributed Data Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual databases face challenges in inheriting indexes for sorting, searching, and aggregation of real tabular data, leading to long processing times and high storage requirements, especially when dealing with large volumes of distributed data.
Innovation Solution
A technique that constructs virtual tabular data with automatically inherited indexes using a D5A file format, which provides indexes for all columns, enabling high-speed sorting, searching, and aggregation, and requires minimal storage space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If subqueries are executed on backend distributed databases to create virtual tabular data, then the virtual database can provide flexible data access, but the processing time becomes excessively long
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing aggregation results, sorted data, and search indexes in the virtual database layer before they are needed. When queries are executed, the system retrieves pre-prepared results rather than computing them on-demand from distributed backend databases, dramatically reducing processing time while maintaining query flexibility.
Solution Approach 2:
The patent segments the database system into multiple layers: the virtual database layer that handles complex queries and aggregations, and the backend distributed database layer that stores raw data. By dividing responsibilities and pre-processing data at the virtual layer, the system resolves the contradiction between flexible access and processing speed.
2Device complexity
If indexes are not inherited for virtual tabular data, then the virtual database structure remains simple, but sorting, searching, and aggregation operations become inefficient
Solution Approach 1:
The patent applies copying by creating index structures in the virtual database layer that mirror or copy the indexing capabilities of the backend databases. These copied indexes enable efficient sorting, searching, and aggregation operations on virtual tabular data without requiring complex transformations of the underlying data storage structure.
Solution Approach 2:
The patent adds another dimension to the database architecture by introducing a virtual database layer with its own index structures, separate from the physical database layer. This dimensional addition enables efficient data operations on virtual tables without modifying the physical storage structure, resolving the contradiction between structural simplicity and operational efficiency.
3Quantity of substance
If data is stored in distributed backend databases, then the system can handle large volumes of data, but storage requirements and data access complexity increase
Solution Approach 1:
The patent introduces the virtual database as an intermediary layer between the user and the distributed backend databases. This intermediary handles complex join operations, data aggregation, and query optimization, simplifying data access for users while maintaining the ability to handle large volumes of distributed data through the backend systems.
Data Source
AI summary
With respect to a data processing method performed by a computer, the data processing method includes constructing an index on a virtual column defined by a predetermined mapping from one or more source columns included in one or more tabular data, by referring to a list of values in the one or more source columns and occurrence counts of the values.