Adaptive Column Storage Layout for Lower Query Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current columnar storage methods in database systems result in low query efficiency due to the need to open multiple files simultaneously for filtering access, especially when sparse data and non-adaptive storage are involved, leading to inefficiencies in handling big data queries.
Innovation Solution
A data storage method that optimizes table data storage by determining candidate and target column combinations based on query requests, merging frequently accessed column data into a single file to minimize query overhead, thereby improving query efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If columnar storage stores each column in a separate file, then data organization is simplified and storage is easy to implement, but multiple files need to be opened simultaneously for querying, resulting in low query efficiency
Solution Approach 1:
The patent merges multiple columns that are frequently accessed together into a single file, creating a hybrid storage structure. This resolves the contradiction by combining the simplicity of separate column storage with the efficiency of merged data access, reducing the number of file openings needed during queries while maintaining organized data structure.
Solution Approach 2:
The patent dynamically adjusts the storage structure by identifying frequently accessed column combinations and merging them based on query patterns. This dynamic adaptation allows the system to optimize for query efficiency while maintaining the underlying columnar organization, resolving the static trade-off between storage simplicity and query performance.
2Device complexity
If traditional columnar storage uses fixed storage structure, then storage management is simple, but it cannot adapt to different application data access characteristics, leading to suboptimal query performance
Solution Approach 1:
The patent performs preliminary analysis of query requests to identify frequently accessed column combinations before executing queries. By pre-processing query patterns and determining optimal column merges in advance, the system adapts to application-specific access characteristics without adding complexity to the storage management process during query execution.
Solution Approach 2:
The patent uses query request feedback to dynamically determine which columns should be merged together. By analyzing actual query patterns and using this feedback to adjust the storage structure, the system adapts to different application data access characteristics while maintaining manageable complexity through automated feedback-driven optimization.
3Productivity
If columnar storage merges all columns into one file, then query efficiency is maximized by reducing file openings, but data organization becomes complex and storage management difficulty increases
Solution Approach 1:
The patent segments columns into different groups based on their access patterns, merging only those columns that are frequently accessed together while keeping other columns separate. This segmentation approach maximizes query efficiency for common query patterns while avoiding the complexity of merging all columns into a single file, as the storage structure remains organized into logical groups.
Data Source
AI summary
This application provides a data storage method and apparatus, a device, a computer-readable storage medium, and a computer program product. The method includes acquiring a plurality of query requests for an application, and parsing the query requests to obtain at least one piece of column information in the query requests; determining candidate column combinations of tables related to the query requests; determining target column combinations of corresponding tables based on a plurality of candidate column combinations belonging to the same table, a total query overhead for executing the plurality of query requests being minimum when the table data of the tables is stored according to respective target column combinations; and storing the table data of the tables according to the target column combinations.