Operator-Based Data Formats for Faster Queries in Unified Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in data processing due to the impact of data format, particularly when data is stored in row-based format, leading to prolonged query times for column-based operations and increased storage layout costs from multiple storage engines.
Innovation Solution
A method and system that adjusts data formats in real-time based on processing needs, using a single storage engine to optimize data processing efficiency by storing data in formats suitable for specific operations, such as row-based, column-based, or hybrid formats, eliminating the need for redundant execution engines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in row-based format, then data processing functions such as aggregation and filtering can be realized, but query execution time is extended and efficiency is reduced when querying a single column of data
Solution Approach 1:
The patent implements dynamic data format conversion by transforming data from row-based format to column-based format based on the type of operation being performed. The system automatically selects the appropriate data format (row-based for row operations, column-based for column operations) and converts the data accordingly, making the data storage format adaptive rather than static. This resolves the contradiction by allowing the system to optimize for either processing efficiency or query speed depending on the operational context.
Solution Approach 2:
The patent changes the data storage format parameter from a fixed row-based structure to a flexible structure that can switch between row-based and column-based formats. By modifying this fundamental parameter based on operational requirements, the system achieves both efficient data processing (through column-based format for aggregations and filters) and fast column querying (through native column-based storage), eliminating the need to choose one format over the other.
2Productivity
If multiple storage engines are used to support different data formats, then data processing efficiency is improved, but device complexity and storage layout costs increase
Solution Approach 1:
The patent makes a single storage engine universal by enabling it to store and manage data in both row-based and column-based formats. The storage engine is designed with format conversion capabilities, allowing it to adapt to different operational requirements without requiring separate specialized engines. This resolves the contradiction by consolidating multiple functions into one engine, reducing system complexity while maintaining the ability to optimize for different processing scenarios.
Solution Approach 2:
The patent introduces a format conversion layer as an intermediary between the storage engine and the data processing operations. This intermediary component handles the transformation between row-based and column-based formats, allowing the single storage engine to serve multiple purposes. The conversion layer mediates between the fixed storage structure and the variable operational requirements, enabling efficient data processing without needing multiple specialized storage engines.
Data Source
AI summary
Described provides a data processing method, where the method includes the following steps: receiving a data processing request; storing the plurality of first data as a first data format and executing the first data processing on the plurality of first data, wherein the first data format is determined based on the first data processing; and storing the plurality of second data as a second data format and executing the second data processing on the plurality of second data, wherein the second data format is determined based on the second data processing. As far as the data can be stored in different formats according to different data processing, the match between data format and data processing are improved. Described further provides a data processing system, as well as an apparatus, a readable storage medium to which the foregoing method is applied.