Dynamic Data Service Load Sharing in Software Defined Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software defined storage (SDS) systems face inefficiencies in implementing inline data services like deduplication and compression, leading to overloading of client nodes and unnecessary network traffic due to duplicate task execution and lack of global balancing.
Innovation Solution
Implementing a method where deduplication and compression are dynamically shared between client and server modules, with fingerprint calculations and data processing distributed across both, allowing for flexible configuration and load-sharing to optimize performance and reduce network traffic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data services (deduplication and compression) are performed only at client modules, then data processing can be done close to data source, but client nodes become overloaded and performance degrades
Solution Approach 1:
The patent segments the data service processing workload between client modules and server modules. Client modules perform initial fingerprint calculations and data services on their local data, while server modules perform fingerprint calculations and data services on data received from multiple clients. This segmentation distributes the computational load and prevents client node overload.
Solution Approach 2:
The patent merges the data service processing capabilities into both client and server modules, creating a distributed processing architecture. Instead of concentrating all processing at the client level, the system combines processing functions at both ends, allowing load sharing and global balancing of computational tasks across the storage network.
2Ease of operation
If data services are performed at each client node independently, then local data can be processed, but duplicate tasks are executed across the network increasing overall traffic
Solution Approach 1:
The patent segments the fingerprint calculation and data service tasks between clients and servers based on data location and processing needs. Clients calculate fingerprints for their local data and perform data services, while servers calculate fingerprints for received data and perform data services, eliminating duplicate processing of the same data across the network.
Solution Approach 2:
The server modules act as intermediaries that receive data from clients, perform centralized fingerprint calculations and data services, and manage the storage operations. This intermediary role prevents duplicate independent processing at each client node and coordinates the data service operations across the distributed system, reducing redundant network traffic.
3Stability of the object's composition
If all data processing is centralized at server modules, then global coordination is achieved, but network traffic increases and processing latency increases
Solution Approach 1:
The patent segments processing tasks so that clients perform fingerprint calculations and data services on their local data before transmission, while servers perform these operations on received data. This segmentation allows parallel processing at multiple levels, reducing the time data spends in transit and minimizing processing latency while maintaining global coordination through the server's centralized management of storage operations.
Data Source
AI summary
A method for processing a data request in a software defined storage system, wherein the software defined storage system comprises one or more nodes configured as a set of client modules operatively coupled to a set of server modules, comprises the following steps. A data request with a data set is received at one of the set of client modules. One or more data services (e.g., deduplication and/or data compression) are performed on the data set, wherein the performance of the one or more data services on the data set is dynamically shared between one or more of the set of client modules and one or more of the set of server modules.