Distributed Column Chunk Data Store Server Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems fail to scale well for data mining and business intelligence applications due to inefficient data storage and retrieval, particularly with large volumes of data, and lack of semantic relationships between data and files, leading to slow processing speeds and inadequate redundancy for handling failures.
Innovation Solution
A distributed column chunk data store system that partitions data tables into column chunks, allowing flexible distribution across multiple storage servers with redundancy support, using a storage services manager for metadata management, compression, and parity calculation, enabling efficient data retrieval and storage server removal.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed storage systems store large volumes of data across multiple servers, then data capacity increases, but retrieval efficiency deteriorates due to lack of semantic relationships between data and files
Solution Approach 1:
The patent segments data into columnar chunks organized by semantic meaning rather than storing complete rows across servers. This allows the system to maintain large data capacity while improving retrieval efficiency by querying only specific column chunks needed for the query, avoiding unnecessary data transfer and processing.
Solution Approach 2:
The patent introduces a metadata layer that acts as an intermediary between storage servers and query processors. This metadata layer maintains semantic relationships between data and files, enabling efficient query routing to the appropriate column chunks without requiring full data retrieval, thus resolving the contradiction between large capacity and efficient retrieval.
2Reliability
If distributed storage systems retrieve rows with many columns, then complete data sets are obtained, but processing efficiency deteriorates when only one or few columns are needed
Solution Approach 1:
The patent segments data storage at the column level rather than row level, organizing data into independent column chunks. This allows the system to retrieve only the specific columns needed for processing while maintaining data completeness for those columns, eliminating the inefficiency of retrieving entire rows when only a few columns are required.
Solution Approach 2:
The patent applies local quality by storing and retrieving data with varying levels of detail based on query requirements. Different column chunks can be retrieved with appropriate granularity, allowing the system to optimize processing efficiency by retrieving only the necessary data portions while maintaining complete and accurate data for the requested columns.
3Adaptability or versatility
If storage servers are removed from distributed systems, then system flexibility improves, but data integrity deteriorates due to loss of redundancy
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing parity information for column chunks before data is lost. When a storage server needs to be removed, the system can reconstruct the lost data using the pre-stored parity information and remaining column chunks, allowing flexible server removal while maintaining data integrity through proactive redundancy management.
Data Source
AI summary
An improved system and method for removing a storage server in a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server provided may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Any data table may be flexibly partitioned into column chunks using one or more columns with various partitioning methods. Storage servers may then be removed and column chunks may be redistributed among the remaining storage servers in the column chunk data store.