ML Data Placement in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face challenges in minimizing latency due to the physically dispersed nature of cloud infrastructure, as conventional replication methods are costly and inefficient, especially when data access requirements are not geographically uniform, and patterns of requests for information may vary over time.
Innovation Solution
A computer-implemented method using a machine learning mapping model to determine primary storage locations based on metadata values associated with prior location requests, which identifies the most suitable data store for data placement within a distributed system, considering geographic dependencies and variations in data usage patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data replication is used to ensure geographic proximity and reduce latency, then data access speed is improved, but storage costs and system complexity increase
Solution Approach 1:
The patent applies local quality by placing data copies selectively in specific geographic locations based on predicted demand patterns rather than uniformly replicating everywhere. The system analyzes metadata from location requests to determine which regions are likely to need particular data, then replicates only to those locations, optimizing the balance between access speed and storage costs.
Solution Approach 2:
The system performs preliminary action by predicting future data access patterns using metadata from historical location requests before actual access occurs. This allows proactive data placement in anticipated high-demand regions, reducing latency without requiring full replication across all possible locations.
2Reliability
If full data replication is implemented across all regions, then data availability is improved, but storage costs increase
Solution Approach 1:
The system implements local quality by tailoring data replication strategies to specific regional demands. Instead of uniform replication, it uses metadata analysis to identify which regions are likely to need which data, then replicates selectively to those locations, maintaining high availability where needed while reducing unnecessary storage costs in low-demand regions.
Solution Approach 2:
The patent applies partial action by implementing selective replication rather than complete replication across all regions. The system determines the optimal subset of regions for data placement based on predicted demand, achieving sufficient availability without the excessive storage costs of universal replication.
3Productivity
If data is stored in high-traffic locations, then average system performance is improved, but geographic specificity of data requirements is not addressed, resulting in poor performance in lower-traffic locations
Solution Approach 1:
The system resolves this contradiction by applying local quality - it analyzes metadata from location requests to understand geographic patterns of data access and tailors data placement to specific regional needs. This allows the system to optimize for local geographic demands rather than relying solely on overall traffic volume, improving both average performance and geographic adaptability.
Solution Approach 2:
The patent applies dynamics by making data placement adaptive and flexible rather than static. The system continuously monitors location request patterns and adjusts data placement decisions based on evolving geographic demand patterns, allowing it to respond to changing traffic distributions and maintain performance across varying conditions.
4Ease of operation
If conventional replication methods are used, then data accessibility is improved, but operational costs increase
Solution Approach 1:
The system reduces operational costs through local quality by analyzing metadata to identify specific high-demand regions and placing data only in those locations rather than using expensive global replication. This selective approach maintains data accessibility where it matters most while significantly reducing storage and data transfer costs.
Solution Approach 2:
The patent applies preliminary action by using metadata from historical requests to predict future access patterns and pre-position data in likely high-demand regions. This proactive approach improves data accessibility when needed while avoiding the costs of maintaining data in all possible locations.
Data Source
AI summary
A method of determining a primary storage location for a data record in a distributed system comprising a plurality of data stores physically located in corresponding geographic locations, includes initialising a machine learning mapping model using topology information of the distributed system, and determining a set of training feature vectors derived from metadata values associated with prior location requests. The model is trained using the training feature vectors and a corresponding set of target primary storage locations. A location request that includes a plurality of metadata values and is associated with a data record is received, and the metadata values are processed to determine a prediction feature vector comprising a plurality of prediction feature values. The model is executed using the prediction feature vector to identify one data store of the plurality of data stores as the primary storage location for the data record associated with the location request.