Columnar Storage for Nested Data Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face inefficiencies in handling large-scale, nested data structures, particularly in reconstructing record structures from columnar storage and performing queries without loading data into a database, which leads to increased computational costs and time consumption.
Innovation Solution
The system employs a columnar storage format for nested data, using repetition and definition levels to efficiently reconstruct records and perform queries directly on columnar data, allowing for in situ data processing without the need for record assembly or database loading.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in columnar format for efficient querying, then query performance is improved, but the complexity of reconstructing nested record structures increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing repetition levels and definition levels in the columnar storage structure. These levels are calculated in advance during data insertion and stored metadata, so that during query operations, the system can directly use these pre-computed values to reconstruct nested record structures without performing complex real-time calculations. This resolves the contradiction by shifting the computational complexity from query time to data insertion time.
2Adaptability or versatility
If data is loaded into a database for querying, then query flexibility is improved, but computational cost and time consumption increase
Solution Approach 1:
The patent extracts the essential querying capabilities directly from the columnar storage structure itself, eliminating the need to load data into a traditional database management system. By embedding repetition levels and definition levels in the columnar storage format, the system enables flexible querying operations to be performed in situ on the stored data, thereby removing the time-consuming data loading step while preserving query flexibility.
3Ease of manufacture
If traditional row-based storage is used, then record structure maintenance is simplified, but query efficiency on large datasets deteriorates
Solution Approach 1:
The patent changes the fundamental storage parameter from row-based to columnar organization, while compensating for the increased complexity by introducing repetition levels and definition levels. These parameters enable the system to efficiently handle nested structures in columnar format by providing metadata that guides reconstruction, thereby achieving both query efficiency on large datasets and manageable record structure maintenance.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
In general, the subject matter described in this disclosure can be embodied in methods, systems, and program products. A computing system receives a first request to insert one or more first data values into a database table. The computing system identifies that first data stored by the database table is stored in a first logical partition of a logical collection of data. The logical collection of data is designated for replication among multiple data centers such that a copy of the logical collection of data is designated to be stored by each of the multiple data centers. The logical collection of data is logically partitioned into multiple logical partitions which together comprise the logical collection of data. The computing system sends, by the computing system and to the first data center, the first request to insert the one or more first data values into the database table.