Server Distribution Pattern for Coded Data Reliability and Security
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies in memory usage and security concerns when duplicating data across multiple servers, particularly when third-party servers are involved, as they can lead to memory inefficiencies and risks of data access and reverse-engineering.
Innovation Solution
A system and method that utilizes error-correcting code schemes to transform raw data into distributable data, which is then distributed across multiple servers in a pattern that maximizes system reliability and entropy, ensuring no single server has the entire data and allowing for data reconstruction and redistribution in case of server failure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is duplicated in full on each individual backup server, then data availability is improved, but memory efficiency deteriorates
Solution Approach 1:
The patent segments data into multiple shards and distributes them across different servers. Instead of duplicating entire datasets on each server, the system divides data into portions and places them on multiple servers, achieving both redundancy and space efficiency through intelligent segmentation and distribution strategies.
Solution Approach 2:
The patent implements nested data structures where metadata about data locations, shards, and recovery information is embedded within the data storage system itself. This allows the system to track and manage distributed data portions efficiently while maintaining the ability to reconstruct complete datasets.
2Quantity of substance
If data is segmented across multiple servers, then memory efficiency is improved, but system complexity deteriorates
Solution Approach 1:
The patent introduces a data distributor as an intermediary component that manages the complexity of data segmentation and distribution. This mediator handles shard creation, assignment to servers, tracking of data locations, and coordination of recovery operations, thereby simplifying the overall system architecture despite the distributed nature of data storage.
Solution Approach 2:
The patent employs configurable parameters such as shard size, number of replicas, and distribution policies that can be adjusted to optimize the balance between memory efficiency and system complexity. By allowing dynamic parameter changes, the system can adapt to different storage requirements and server configurations without fundamental architectural changes.
3Quantity of substance
If error-correcting code schemes are used to transform data, then storage efficiency is improved, but data reconstruction complexity deteriorates
Solution Approach 1:
The patent uses error-correcting code schemes that create redundant copies and transformed versions of data portions. These coded representations are distributed across servers, allowing the original data to be reconstructed even if some portions are lost or corrupted. The copying mechanism includes both direct replicas and encoded transformations.
Solution Approach 2:
The patent transforms data by changing its representation through error-correcting code schemes, converting original data into coded forms that are more resilient to loss. This parameter change in data representation enables efficient storage with reduced redundancy while maintaining the ability to reconstruct original data through decoding operations.
Data Source
AI summary
Described are a system, method, and computer program product for generating a data storage server distribution pattern. The method includes determining a set of servers and raw data to be stored. The method also includes transforming the raw data according to an error-correcting code scheme to produce distributable data. The method further includes determining a server reliability of each server in the set of servers. The method further includes generating the data storage server distribution pattern based on maximizing a system reliability relative to maximizing a system entropy. System reliability may be based on a minimum reliability of the set of servers, and system entropy may be based on a cumulated information entropy of each server of the set of servers. The method further includes distributing the distributable data to be stored across at least two servers of the set of servers according to the data storage server distribution pattern.