Storage Performance Optimization via Segmented ESS Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data warehousing and Data Mart environments face challenges in balancing security and performance, particularly in high-data-volume or IO-intensive applications, where single storage configurations either incur excessive costs for security or compromise on data throughput and scalability.
Innovation Solution
A system that connects multiple servers with local storage to a centralized External Storage System (ESS), maintaining a full reference copy of data for high availability and security, while allowing dynamic reprovisioning and supporting high performance failover and disaster recovery, and optimizing IO performance across mixed storage components.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If an External Storage System (ESS) is used to guarantee security, then reliability is improved, but cost increases prohibitively
Solution Approach 1:
The storage system is segmented into two distinct layers: a centralized ESS layer for security-critical functions (backup, disaster recovery, compliance archiving) and local storage layers for performance-critical operations (active data access, caching, high-speed I/O). This segmentation allows each layer to fulfill its specific role efficiently, preventing the need to provision the entire system at ESS cost levels while maintaining security where needed.
Solution Approach 2:
Different storage qualities are applied to different data and operations based on their requirements. Local storage systems provide high-speed access for frequently accessed data, while the ESS provides secure, compliant storage for archival and backup data. This local quality approach ensures that security requirements are met for appropriate data without incurring ESS-level costs for all storage operations.
2Productivity
If local storage systems are used for high data throughput, then productivity is improved, but reliability deteriorates due to inability to store high data volumes effectively
Solution Approach 1:
The storage architecture segments data based on access patterns and requirements: hot data requiring high throughput is stored locally on fast storage systems, while cold data requiring high capacity is stored on the ESS. This segmentation allows the system to achieve both high throughput for active operations and high capacity for archival storage without compromise.
Solution Approach 2:
The system employs storage virtualization and data management software as intermediaries that transparently manage data placement between local and remote storage. These intermediaries handle data movement, caching strategies, and access routing, allowing local storage to deliver high throughput while the ESS provides the underlying capacity for high data volumes.
3Reliability
If ESS is used for centralized high-availability services, then reliability is improved, but data throughput is reduced
Solution Approach 1:
The system segments storage functions by separating high-availability services (backup, disaster recovery, compliance) from high-throughput operations (active data access, reporting, analytics). Local storage systems handle throughput-critical workloads independently, while the ESS handles availability-critical services, eliminating the throughput bottleneck that would result from using ESS for all operations.
Solution Approach 2:
High-availability services are provided with appropriate quality levels: the ESS provides robust, secure backup and recovery capabilities, while local storage provides high-speed access for active operations. This local quality approach ensures that availability requirements are met without forcing all data access through the slower ESS path.
4Productivity
If local storage is used for each server, then productivity is improved, but device complexity increases due to need for sophisticated failover recovery solutions
Solution Approach 1:
The system merges previously separate functions into a unified architecture: local storage handles active data access while the centralized ESS consolidates backup, disaster recovery, and compliance functions. This consolidation reduces the complexity of managing distributed backup solutions across multiple servers while maintaining high local access speeds for productive operations.
Data Source
AI summary
A system and method for enhancing data throughput in data warehousing environments by connecting multiple servers having local storages with designated external storage systems, such as, for example, those provided by SANS. The system and method may preserve a full reference copy of the data in a protected environment (e.g., on the external storage system) that is fully available. The system and method may enhance overall I/O potential performance and reliability for efficient and reliable system resource utilization.


