Distributed Column Chunk Data Store Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems fail to scale well for data mining and business intelligence applications, particularly when handling large volumes of data, due to inefficient storage and retrieval processes, slow processing speeds during transactions, and lack of semantic operations in distributed file systems.
Innovation Solution
A distributed column chunk data store system that partitions data tables into column chunks and distributes them across multiple storage servers, using a database engine with modules for loading, querying, metadata management, and transaction services, along with storage services manager for striping and compressing data, allowing flexible partitioning methods and redundancy policies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed databases store horizontally partitioned tables with all columns for each row, then data can be stored across multiple servers, but retrieval efficiency deteriorates because queries must retrieve all columns even when only one or few are needed
Solution Approach 1:
The patent segments data storage by organizing it into columnar format where data is divided by columns rather than rows. Each column is stored separately, allowing the system to retrieve only the specific columns needed for a query without fetching entire rows. This segmentation resolves the contradiction by enabling efficient partial data retrieval while maintaining distributed storage capacity.
2Device complexity
If distributed file systems provide only low-level storage primitives, then implementation is simple, but semantic operations for data storage and retrieval are unavailable
Solution Approach 1:
The patent implements a multi-functional system that combines low-level storage primitives with high-level semantic operations. The columnar storage engine provides both basic file operations and advanced capabilities including query processing, data redistribution, storage replacement, and dynamic storage addition. This universal approach resolves the contradiction by delivering both simplicity and versatility.
3Reliability
If transaction processing systems rollback database on failed transactions, then data integrity is maintained, but processing speed deteriorates for large data set updates
Solution Approach 1:
The patent extracts the rollback mechanism from traditional transaction processing and replaces it with a columnar storage architecture that inherently maintains data integrity through its structure. The system processes large data sets by working with columnar chunks that can be independently managed, eliminating the need for complete database rollbacks while maintaining reliability. This extraction resolves the contradiction by preserving data integrity without the performance penalty of full rollbacks.
Data Source
AI summary
An improved system and method for a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Any data table may be flexibly partitioned into column chunks using one or more columns as a key with various partitioning methods. There may also be a storage policy for specifying how to partition a data table for distributing column chunks across multiple servers and for specifying a level of redundancy for recovery from failure of storage servers.