Distributed Column Chunk Data Store Query Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed storage systems fail to scale well for data mining and business intelligence applications due to inefficient data storage and retrieval, particularly with large volumes of data, and lack of semantic operations for data management.

Innovation Solution

A distributed column chunk data store system that partitions data tables into column chunks and distributes them across multiple storage servers, using a storage services manager for striping and compression, and query processing servers for optimized query execution across a hierarchy of servers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If distributed storage systems store large volumes of data across multiple servers, then data capacity increases, but query processing efficiency deteriorates due to inefficient data retrieval mechanisms

Engineering Contradiction:

Improvedata capacityVSAvoidquery processing efficiency

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent segments data tables into column chunks and distributes them across multiple storage servers. This segmentation allows queries to access only specific column chunks rather than entire tables, improving query processing efficiency while maintaining large data capacity across the distributed system.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements local quality by creating column-oriented storage where each server stores specific columns of data. This allows query processing to focus on relevant data portions locally, avoiding unnecessary data retrieval from other servers, thus improving query efficiency without sacrificing overall data capacity.

Inventive Principle:

Principle #3Local quality

2Adaptability or versatility

If distributed databases store horizontally partitioned tables across servers, then data scalability improves, but storage and retrieval efficiency deteriorates due to retrieving unnecessary rows

Engineering Contradiction:

Improvedata scalabilityVSAvoidstorage and retrieval efficiency

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent segments data at the column level rather than row level, creating column chunks that can be independently stored and retrieved. This segmentation enables the system to scale data horizontally while retrieving only the specific columns needed for each query, eliminating the inefficiency of retrieving entire rows.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent inverts the traditional row-oriented storage approach by implementing column-oriented storage. Instead of storing and retrieving complete rows, the system stores and retrieves specific columns, which fundamentally changes the efficiency characteristics while maintaining scalability.

Inventive Principle:

Principle #13The other way round (Inversion)

3Reliability

If traditional distributed databases process transactions with rollback capability, then data integrity is maintained, but processing speed deteriorates for large data sets

Engineering Contradiction:

Improvedata integrityVSAvoidtransaction processing speed

Core Design Contradiction:

ReliabilityVSSpeed

Solution Approach 1:

The patent segments transactions into column-level operations rather than row-level operations. This allows the system to process and commit only the affected columns rather than rolling back entire transactions, maintaining data integrity while significantly improving processing speed for large data sets.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent extracts the rollback mechanism from the traditional transaction processing model by implementing column-level commit capabilities. This extraction allows the system to commit successful column operations independently, eliminating the need for complete transaction rollback and thereby improving processing speed while maintaining integrity.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS7921131B2Method using a hierarchy of servers for query processing of column chunks in a distributed column chunk data store

Publication Date: 2011.04.05 GOOGLE LLC

AI summary

An improved system and method for query processing in a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server provided may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Query processing may be performed by storage servers or query processing servers operably coupled by a network to storage servers in the column chunk data store. To do so, a hierarchy of servers may be dynamically determined to process execution steps of a query transformed for distributed processing.