Distributed Column Chunk Data Store Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems fail to scale well for data mining and business intelligence applications due to inefficient data retrieval and processing of large volumes, particularly with terabytes of data, and lack semantic operations for data storage and retrieval.
Innovation Solution
A distributed column chunk data store system that partitions data tables into column chunks, distributes them across multiple storage servers, and uses a storage services manager for striping and compression, along with query processing servers that dynamically determine a hierarchy of servers for distributed query processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed storage systems store large volumes of data across multiple servers, then data capacity increases, but data retrieval efficiency deteriorates
Solution Approach 1:
The patent segments data tables into column chunks that can be independently stored and retrieved across multiple servers. This allows the system to maintain large data capacity while improving retrieval efficiency by only fetching relevant column chunks rather than entire tables, directly resolving the contradiction between storage capacity and retrieval efficiency.
2Loss of information
If distributed databases retrieve rows with many columns, then complete data is obtained, but retrieval efficiency deteriorates when only few columns are needed
Solution Approach 1:
The patent extracts only the necessary column chunks required to answer a query, rather than retrieving entire rows with all their columns. This extraction mechanism eliminates unnecessary data transfer while maintaining data completeness for the specific query, resolving the contradiction between information completeness and retrieval efficiency.
3Quantity of substance
If traditional distributed databases are used for data mining applications, then data storage is achieved, but processing speed deteriorates
Solution Approach 1:
The patent segments both data storage and processing operations into column chunk units that can be processed in parallel across multiple servers. This segmentation enables simultaneous processing of different column chunks, dramatically improving processing speed for data mining applications while maintaining large storage capacity.
Solution Approach 2:
The patent performs preliminary actions by pre-computing and storing intermediate results of data mining operations in the distributed cache. This preliminary computation reduces the processing speed requirement for subsequent queries, as results can be quickly retrieved from cache rather than重新 computed.
4Ease of manufacture
If distributed file systems provide low-level storage primitives, then storage functionality is achieved, but semantic operations for data retrieval are lacking
Solution Approach 1:
The patent implements a multi-functional system where the distributed storage infrastructure simultaneously provides low-level storage primitives and high-level semantic operations. The column chunk architecture enables both basic file operations and advanced data mining operations like aggregation, filtering, and joining across distributed data, achieving universality that resolves the contradiction between storage functionality and semantic operation capability.
Data Source
AI summary
An improved system and method for query processing in a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server provided may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Query processing may be performed by storage servers or query processing servers operably coupled by a network to storage servers in the column chunk data store. To do so, a hierarchy of servers may be dynamically determined to process execution steps of a query transformed for distributed processing.