Distributed Column Chunk Data Store for Scalable Data Mining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems fail to scale well for data mining and business intelligence applications, as they are inefficient in storing and retrieving large volumes of data, and lack semantic operations for data management and redundancy.
Innovation Solution
A distributed column chunk data store system that partitions data tables into column chunks, using multiple storage servers with a database engine, storage shared memory, and a storage services manager for efficient data distribution, compression, and redundancy management, allowing flexible partitioning and incremental updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed storage systems are used for large volumes of data, then data capacity is improved, but retrieval efficiency and processing speed deteriorate
Solution Approach 1:
The patent segments data tables into column chunks, where each chunk contains data for a specific column across multiple rows. This segmentation allows the system to store large volumes of data efficiently while enabling selective retrieval of only the required columns, thus maintaining high retrieval efficiency even as data capacity scales to terabytes.
Solution Approach 2:
The patent implements local quality by organizing data into column-based chunks where each chunk has specialized structure optimized for its specific column type. This allows different compression techniques and storage strategies to be applied to different column chunks based on their specific characteristics, improving both storage efficiency and retrieval performance for targeted columns.
2Quantity of substance
If traditional distributed databases are used, then data storage capability is improved, but query processing efficiency deteriorates
Solution Approach 1:
The patent extracts only the necessary column chunks required to answer a query from the distributed storage system, rather than retrieving entire tables or rows. This extraction mechanism significantly reduces the amount of data that needs to be transferred and processed, thereby improving query processing speed while maintaining the ability to store large volumes of data across the distributed system.
3Adaptability or versatility
If distributed file systems are used, then storage flexibility is improved, but semantic operations capability deteriorates
Solution Approach 1:
The patent implements a multi-functional distributed storage system that combines the flexibility of distributed file systems with database-like semantic operations. The system provides both file-level access and column-level data manipulation capabilities, enabling semantic operations such as selective column retrieval, filtering, and aggregation while maintaining storage flexibility across distributed nodes.
4Ease of manufacture
If data is stored in row-based format, then simplicity of storage is improved, but retrieval efficiency for specific columns deteriorates
Solution Approach 1:
The patent segments data from row-based storage into column-based chunks, where each chunk contains data for a specific column across multiple rows. This segmentation transforms the storage structure to enable efficient retrieval of specific columns without requiring reading of entire rows, thereby improving retrieval efficiency while maintaining manageable storage complexity through systematic organization.
Data Source
AI summary
An improved system and method for importing update data in a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server provided may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Any data table may be flexibly partitioned into column chunks using one or more columns with various partitioning methods. Update data may then be incrementally imported as separate column chunks that may later be merged with the column chunks of the partitioned data table.