Column Chunk Data Store Recovery via Parity Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems fail to scale well for data mining and business intelligence applications due to inefficient data storage and retrieval, particularly with large volumes of data, and lack fault-tolerance mechanisms.
Innovation Solution
A distributed column chunk data store system that partitions data tables into column chunks, distributes them across multiple storage servers, and uses a storage services manager for redundancy and recovery, including compression and metadata management, to ensure efficient data storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in traditional distributed databases or file systems, then storage capacity can be extended, but retrieval efficiency and scalability for large volumes of data deteriorate
Solution Approach 1:
The patent segments data tables into column chunks, where each chunk contains a subset of columns. This segmentation allows the system to store large volumes of data efficiently while enabling selective retrieval of only the required columns for query processing, thus maintaining high retrieval efficiency even as storage capacity scales to terabytes
Solution Approach 2:
The patent implements local quality by storing different column chunks on different storage servers based on their access patterns and query requirements. Frequently accessed columns are optimized for fast retrieval, while less accessed columns use different storage strategies, allowing the system to handle large data volumes without compromising overall retrieval efficiency
2Quantity of substance
If traditional distributed storage systems are used, then storage capacity can be increased, but fault-tolerance capability deteriorates
Solution Approach 1:
The patent implements preliminary action by pre-computing and storing checksums for each column chunk. When data is written to storage servers, the system immediately calculates and stores verification information, enabling rapid detection of data corruption or loss without requiring complex real-time verification mechanisms
Solution Approach 2:
The patent uses copying by maintaining redundant copies of critical metadata and checksum information across multiple storage servers. This allows the system to tolerate server failures while maintaining data integrity, as verification information can be retrieved from alternative sources if primary storage becomes unavailable
3Quantity of substance
If distributed databases are used for large volumes of data, then storage capacity can be extended, but query processing efficiency deteriorates
Solution Approach 1:
The patent extracts only the necessary column chunks required for query processing from the distributed storage system. Instead of retrieving entire tables or large data sets, the system identifies and retrieves only the specific column chunks containing the required data, dramatically improving query processing efficiency while maintaining the ability to store and manage terabytes of data across distributed servers
Data Source
AI summary
An improved system and method for recovery from failure of a storage server in a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server provided may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Any data table may be flexibly partitioned into column chunks using one or more columns with various partitioning methods. Storage servers may then fail and column chunks may be recreated from parity column chunks and redistributed among the remaining storage servers in the column chunk data store.