Columnar Storage for Nested Data Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management solutions for large-scale analytical data processing, particularly in web companies, face scalability limitations and inefficiencies in handling nested data structures, leading to slow query times and high computational costs due to the need to load entire records for analysis.
Innovation Solution
A scalable, interactive ad-hoc query system is developed using multi-level execution trees and columnar data layout, which decomposes records into column stripes with field values and level information, allowing for efficient assembly and querying of nested records without loading entire records, and includes metadata for supporting various queries and fault tolerance mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional row-oriented data storage is used, then record integrity is maintained, but query performance degrades due to loading entire records
Solution Approach 1:
The patent segments records into columnar blocks where data is stored by column rather than by row. This allows the query system to load only the specific columns needed for a query instead of loading entire records, directly resolving the contradiction between maintaining record integrity and improving query performance.
Solution Approach 2:
The patent transitions from traditional row-oriented storage to column-oriented storage, changing the dimensional organization of data. This dimensional change enables efficient selective loading of data columns, improving query execution speed without compromising data integrity through the use of metadata and block structures.
2Productivity
If columnar storage is implemented, then query efficiency improves, but data structure complexity increases
Solution Approach 1:
The patent divides data into manageable columnar blocks with standardized structures. Each block contains metadata and organized data columns, which simplifies the complexity by creating uniform, reusable units that can be efficiently processed, rather than dealing with unstructured complex data formats.
Solution Approach 2:
The patent changes the organizational parameters of data storage from row-based to column-based structures. This parameter change fundamentally alters how data is accessed and processed, enabling parallel processing and efficient queries while the complexity is managed through standardized block definitions and metadata schemas.
3Reliability
If entire records are loaded for analysis, then data completeness is ensured, but computational resources are wasted
Solution Approach 1:
The patent extracts and loads only the specific columns or data blocks that are relevant to the current query, rather than loading entire records. This extraction approach ensures that the necessary data for analysis is complete while avoiding the waste of computational resources on irrelevant data, directly addressing the contradiction between data completeness and energy efficiency.
Solution Approach 2:
The patent applies partial action by loading only the portion of data that is needed for the current query (specific columns or blocks) rather than loading the entire record. This partial loading strategy maintains reliability for the required data while significantly reducing computational energy consumption by avoiding processing of unnecessary data.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
A computer system accesses a collection of data records. Each record in the collection includes a plurality of data values and a plurality of data elements that identify semantics of corresponding data values from the plurality of data values. One or more of the data records each include multiple instances of a same data element and include data values that correspond to the multiple instances of the same data element. The computer system generates a collection of columnar stripes. The collection of columnar stripes includes the data values from each data record in the collection of data records. Each columnar stripe in the collection of columnar stripes includes all data values that correspond to a specific data element from each of the records in the collection of records.