Distributed Data Storage System with Redundant Server Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems struggle to efficiently handle large volumes of data with high write and access speeds while maintaining redundancy, often becoming costly and bogging down under heavy loads, especially in large-scale applications like spam email filtering systems.
Innovation Solution
A data storage and retrieval system comprising multiple data processing servers, data storage servers, and a key manager, which employs redundant data storage across TCP/IP connections, allowing data to be written to multiple storage units and utilizing data spooling and master/slave configurations for efficient data management and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional systems use large databases with file systems on RAID systems, then data storage capacity is improved, but system cost increases and performance degrades under heavy loads
Solution Approach 1:
The patent divides the storage system into multiple independent storage servers (e.g., storage servers 204, 206, 208, 210) that can be distributed across different physical locations. Each storage server handles a portion of the data storage workload independently, allowing the system to scale capacity without creating a single point of failure or performance bottleneck. This segmentation enables parallel data operations and improves overall write and access speeds.
Solution Approach 2:
The patent introduces a hierarchical storage architecture with multiple layers: storage servers connected via network, domain name system servers for routing, and load balancers for distribution. This multi-dimensional approach allows data to be accessed through multiple paths and routes, improving performance by distributing I/O operations across different network and storage resources rather than relying on a single RAID array.
2Reliability
If conventional systems use expensive RAID systems, then reliability is improved, but system cost increases significantly
Solution Approach 1:
The patent implements redundant data storage by writing copies of data to multiple storage servers. The system maintains data copies across different storage locations and uses load balancers to distribute and manage these copies. This copying approach provides fault tolerance and reliability without requiring expensive RAID hardware, as standard storage devices can be used in a distributed configuration.
Solution Approach 2:
The patent introduces load balancers (e.g., load balancer 202) as intermediary components that manage data distribution and retrieval across multiple storage servers. These intermediaries handle the complexity of redundancy management, failover, and load distribution, providing enterprise-level reliability through software-based solutions rather than expensive hardware RAID controllers.
3Reliability
If data is stored redundantly across multiple servers, then reliability is improved, but device complexity increases
Solution Approach 1:
The patent employs domain name system servers (e.g., DNS server 203) that perform multiple functions: routing data requests to appropriate storage servers, managing data location information, and providing load balancing capabilities. This multi-functional approach reduces the need for separate specialized components for each function, simplifying the overall system architecture while maintaining redundancy and reliability.
4Quantity of substance
If the system handles large numbers of data blocks, then data storage capacity is improved, but system performance degrades
Solution Approach 1:
The patent divides incoming data streams into multiple data blocks that are distributed across different storage servers simultaneously. Each storage server processes and stores data blocks in parallel, allowing the system to handle large volumes of data without sequential processing bottlenecks. This segmentation maintains high throughput even as the number of data blocks increases.
Solution Approach 2:
The patent implements continuous data flow processing where load balancers and storage servers operate in continuous parallel streams. Data is routed, stored, and managed without interruption across multiple servers simultaneously, maintaining consistent high-speed performance regardless of the volume of data blocks being processed. The system avoids batch processing delays by maintaining continuous operational throughput.
Data Source
AI summary
Data storage and retrieval systems and related methods are disclosed herein. In one aspect, a system includes data processing servers configured to receive incoming data and transmit the data for storage. The system also includes data storage servers each coupled to one or more data storage units and configured to receive transmitted data for writing to the one or more data storage units, and to read data from the one or more data storage units. Furthermore, the systems may include data retrieval servers coupled to one or more of the data storage servers and configured to retrieve data from the one or more data storage units. The system further includes process modules each associated with one of the data storage servers, where at least two of the process modules are configured to write a portion of the data to corresponding data storage units. In addition, each of the process modules are configured to transmit an acknowledgment associated with each of the corresponding at least two data storage units upon the writing of the data portion in the at least two data storage units. Furthermore, some embodiments may include a key manager coupled to the data processing servers and configured to receive keys that indicate the location of the data portions in the one or more data storage units. Additionally, each data retrieval server may be coupled to the key manager and employ the keys to retrieve data from the one or more data storage units.