Distributed Column Chunk Data Store Query Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed storage systems fail to scale well for data mining and business intelligence applications due to inefficient data retrieval and processing of large volumes, particularly in handling terabytes of data, and lack semantic operations for data storage and retrieval.

Innovation Solution

A distributed column chunk data store system that partitions data tables into column chunks, distributes them across multiple storage servers, and uses a storage services manager for striping and compression, along with query processing servers that dynamically determine a hierarchy of servers for distributed query processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If distributed storage systems store large volumes of data across multiple servers, then data capacity increases, but data retrieval efficiency deteriorates

Engineering Contradiction:

Improvedata capacityVSAvoiddata retrieval efficiency

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent segments data tables into column chunks and distributes them across multiple storage servers. This segmentation allows the system to scale data capacity while maintaining retrieval efficiency by enabling parallel access to specific column chunks rather than retrieving entire tables.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements columnar storage where data is organized by columns rather than rows, allowing queries to retrieve only the specific columns needed. This local quality optimization ensures that retrieval efficiency is maintained by transferring and processing only relevant data portions from distributed servers.

Inventive Principle:

Principle #3Local quality

2Quantity of substance

If distributed databases store horizontally partitioned tables on multiple servers, then data capacity increases, but storage and retrieval efficiency deteriorates

Engineering Contradiction:

Improvedata capacityVSAvoidstorage and retrieval efficiency

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent segments data into column chunks distributed across servers, replacing traditional horizontal row partitioning. This columnar segmentation improves efficiency by allowing selective retrieval of specific columns rather than entire rows, reducing I/O operations and processing overhead.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent inverts the traditional row-oriented storage model by implementing columnar storage. This inversion allows the system to store and retrieve data in the most efficient format for analytical queries, where columnar data can be compressed and processed more efficiently than row-based structures.

Inventive Principle:

Principle #13The other way round (Inversion)

3Loss of information

If distributed storage systems retrieve rows with many columns, then complete data is retrieved, but retrieval efficiency deteriorates when only one or few columns are needed

Engineering Contradiction:

Improvedata completenessVSAvoidretrieval efficiency

Core Design Contradiction:

Loss of informationVSProductivity

Solution Approach 1:

The patent extracts and retrieves only the specific columns needed for query processing from the distributed column chunks, rather than retrieving entire rows with all columns. This extraction principle maintains data completeness for the required information while dramatically improving retrieval efficiency by reducing data transfer and processing volume.

Inventive Principle:

Principle #2Taking out (Extraction)

4Quantity of substance

If traditional distributed databases are used for data mining applications, then data storage is achieved, but processing speed deteriorates

Engineering Contradiction:

Improvedata storage capacityVSAvoidprocessing speed

Core Design Contradiction:

Quantity of substanceVSSpeed

Solution Approach 1:

The patent segments data into column chunks distributed across multiple servers, enabling parallel processing of data mining operations. Each server can process its assigned column chunks independently and simultaneously, dramatically improving processing speed while maintaining storage capacity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary data organization into columnar format and pre-computes certain aggregations during data loading. This preliminary action prepares the data in an optimized format that accelerates subsequent data mining and business intelligence processing operations.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS7921087B2Method for query processing of column chunks in a distributed column chunk data store

Publication Date: 2011.04.05 GOOGLE LLC

AI summary

An improved system and method for query processing in a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server provided may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Query processing may be performed by storage servers or query processing servers operably coupled by a network to storage servers in the column chunk data store. To do so, a hierarchy of servers may be dynamically determined to process execution steps of a query transformed for distributed processing.