Distributed Column Chunk Data Store Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems fail to scale well for data mining and business intelligence applications due to inefficient data retrieval and processing of large volumes, particularly in handling terabytes of data, and lack semantic operations for data storage and retrieval.
Innovation Solution
A distributed column chunk data store system that partitions data tables into column chunks, using multiple storage servers with a database engine, storage shared memory, and a storage services manager for efficient storage and query processing, allowing flexible partitioning methods and dynamic server hierarchy determination for distributed query processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed storage systems use traditional row-based storage architecture, then data can be stored across multiple servers, but data retrieval efficiency deteriorates when only one or few columns are needed from many columns
Solution Approach 1:
The patent segments data storage at the column level rather than storing complete rows. Each server stores specific column chunks independently, allowing the system to retrieve only the required columns for a query without fetching unnecessary data from other columns. This segmentation enables efficient partial data retrieval and reduces network transfer overhead.
Solution Approach 2:
The patent implements local quality by allowing different servers to store different column chunks with different characteristics. Each server can optimize its stored column chunks locally, and the system can retrieve data from the specific servers that hold the required columns, rather than transferring data across the entire distributed system.
2Quantity of substance
If distributed databases store horizontally partitioned tables on multiple servers, then storage capacity increases, but query processing efficiency decreases due to retrieving unnecessary rows
Solution Approach 1:
The patent segments both storage and query processing along column lines. Data is partitioned into column chunks distributed across servers, and queries are processed by identifying and retrieving only the specific column chunks needed, rather than retrieving entire rows from multiple servers. This maintains storage scalability while improving query efficiency.
Solution Approach 2:
The patent extracts and retrieves only the necessary column chunks from the distributed storage system based on query requirements. Instead of retrieving complete rows that contain all columns, the system extracts precisely the columns needed for processing, reducing data transfer and processing overhead.
3Reliability
If transaction processing systems roll back database on failed transactions, then data integrity is maintained, but processing speed deteriorates for large data sets
Solution Approach 1:
The patent extracts and processes only the specific column chunks affected by a transaction, rather than processing or rolling back entire rows or tables. This selective approach maintains data integrity for the affected columns while significantly reducing the scope of rollback operations, thereby improving transaction processing speed for large data sets.
Solution Approach 2:
The patent applies local quality by performing transaction processing and rollback operations only on the specific column chunks that are affected, rather than applying them globally across all data. This localized approach maintains reliability where needed while minimizing the performance impact of transaction management.
4Adaptability or versatility
If distributed file systems provide low-level storage primitives, then storage flexibility increases, but semantic operations for data management become unavailable
Solution Approach 1:
The patent implements a multi-functional system that combines the storage flexibility of distributed file systems with the semantic operations of database systems. The column chunk architecture provides both low-level storage primitives for flexibility and high-level semantic operations for ease of data management, including distributed query processing, data redistribution, and storage optimization capabilities.
Data Source
AI summary
An improved system and method for query processing in a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server provided may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Query processing may be performed by storage servers or query processing servers operably coupled by a network to storage servers in the column chunk data store. To do so, a hierarchy of servers may be dynamically determined to process execution steps of a query transformed for distributed processing.