Data Allocation Engine for Distributed Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in optimizing user data allocation across heterogeneous servers and storage nodes due to varying user behaviors and hardware characteristics, making it difficult to ensure performance, reliability, and compliance with service policies, especially with dynamic changes in user patterns and hardware capacity.
Innovation Solution
A data engine analyzes user and system characteristics to generate a user assignment plan that balances user characteristics with hardware capabilities, using forecasting and optimization techniques to determine the best allocation and replication strategy across storage nodes, minimizing data movement and cost.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is distributed across heterogeneous storage nodes, then system capacity and performance are maximized, but allocation complexity increases
Solution Approach 1:
The patent transforms the allocation problem from a complex combinatorial optimization into a solvable mathematical program by changing parameters: using forecasted future characteristics instead of current state, applying regularization to handle heterogeneity, and formulating as a convex optimization problem with objective function and constraints that can be efficiently solved
Solution Approach 2:
The system performs preliminary forecasting of user characteristics and storage node capacities before making allocation decisions. By predicting future states rather than reacting to current states, the system proactively optimizes allocations before changes occur, reducing the complexity of real-time decision-making
2Reliability
If user data is replicated for reliability, then system reliability improves, but storage capacity utilization decreases
Solution Approach 1:
The patent applies different replication factors to different storage nodes and user data based on local characteristics. Instead of uniform replication, the system determines optimal replication factors locally for each node-user pair by solving the optimization problem, allowing high-reliability data to be replicated more while low-reliability data uses minimal replication, thus balancing reliability requirements with capacity utilization
3Productivity
If data reallocation is performed frequently to optimize performance, then system performance improves, but data movement cost increases
Solution Approach 1:
The system performs allocations periodically at scheduled intervals rather than continuously or on every state change. This periodic approach allows the optimization engine to compute optimal allocations at discrete times, avoiding the excessive data movement that would result from continuous reallocation while still maintaining performance through regular optimization cycles
Solution Approach 2:
The optimization framework serves multiple functions simultaneously: it determines allocation, calculates migration costs, forecasts future states, and evaluates performance metrics all within a single unified mathematical program. This multi-functionality reduces the need for separate optimization passes and associated data movements
Data Source
AI summary
In a distributed storage system such as those in a data center or web based service, user characteristics and characteristics of the hardware such as storage size and storage throughput impact the capacity and performance of the system. In such systems, an allocation is a mapping from the user to the physical storage devices where data/information pertaining to the user will be stored. Policies regarding quality of service and reliability including replication of user data/information may be provided by the entity managing the system. A policy may define an objective function which quantifies the value of a given allocation. Maximizing the value of the allocation will optimize the objective function. This optimization may include the dynamics in terms of changes in patterns of user characteristics and the cost of moving data/information between the physical devices to satisfy a particular allocation.