Columnar Storage for Nested Data Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management solutions for large-scale analytical data processing, particularly with nested data models, face scalability limitations and inefficiencies in query execution due to the need to load entire records for analysis, leading to high computational costs and slow response times.
Innovation Solution
A data storage and processing service utilizing a columnar storage representation for nested records, combined with a multi-level serving tree and query dispatcher, allows for efficient query execution by decomposing records into column stripes and using finite state machines for assembly, enabling rapid and scalable analysis without loading entire records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional data management solutions load entire records for analysis, then query execution can be performed on complete data, but computational costs increase and response times slow down
Solution Approach 1:
The patent segments records into columnar stripes, organizing data by columns rather than by complete records. This allows the query system to access and process only the specific column data needed for a given query, rather than loading entire records. The segmentation enables selective data access at the column level, directly resolving the contradiction between executing complete queries and maintaining fast response times.
2Loss of information
If traditional solutions extract all data fields from every record, then comprehensive analysis is possible, but processing time and computational resources increase
Solution Approach 1:
The patent extracts only the necessary column data from the stored record structure, rather than extracting all fields from every record. The columnar storage format allows the system to take out and process only the specific columns required by the query, leaving other data untouched. This extraction approach maintains data completeness for the queried fields while dramatically reducing processing time by avoiding unnecessary data extraction.
3Stability of the object's composition
If data is stored in traditional record-oriented format, then data integrity and structure are maintained, but query efficiency on large datasets decreases
Solution Approach 1:
The patent changes the data organization from a record-oriented dimension (rows) to a columnar dimension (columns). This dimensional transformation allows the system to maintain data structure integrity through the columnar format while enabling much faster query execution. The columnar organization creates a new dimension for data access that is optimized for analytical queries, resolving the contradiction between structural stability and query productivity.
4Productivity
If scalable data processing is implemented across thousands of cores, then parallelism increases, but system complexity and coordination overhead increase
Solution Approach 1:
The patent uses a columnar storage format that can be efficiently replicated and distributed across multiple processing cores. Each core can work on identical or different column segments independently, with the replicated columnar structure enabling straightforward parallel processing. This copying approach allows scalable deployment across thousands of cores while minimizing coordination complexity, as each processor handles self-contained column data without requiring complex inter-core coordination.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
In general, the subject matter described in this specification can be embodied in methods systems, and program products. A request to store data is received. The data is stored as an object in a repository. A request to create a table is received, where the request identifies a name for the table. The table is created with the name. A request to import the data into the table is received. The data is imported into the table, where importing the data in the object into the table includes converting the data in the object into columnar stripes, and storing the columnar stripes in association with the table. A request to perform a query on the table is received, where the request includes the query and identifies the table. The query is performed on the table, where performing the query includes querying one or more of the columnar stripes.