Object Storage Access Prediction via History Inversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in access time due to the large volume of data generated, leading to complications in storage and retrieval, particularly as the number of users increases, necessitating improved methods to predict and prefetch frequently accessed data.
Innovation Solution
The system predicts future access requests by inverting historic access patterns, proactively moving data from high-latency storage to low-latency storage using a low-priority process that learns from spare CPU cycles, independent of the number of access nodes, and maintains separate access histories and predicted futures for each object.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in high-latency storage (mechanical hard disk), then storage capacity is increased, but access time increases
Solution Approach 1:
The system performs preliminary actions by predicting future access requests through inverting access histories and proactively pre-fetching identified objects from high-latency storage to low-latency storage before they are actually requested. This advance preparation eliminates the time penalty of high-latency storage when objects are needed.
Solution Approach 2:
The patent introduces low-latency storage as an intermediary layer between high-latency storage and the access nodes. Objects are moved through this intermediate storage layer that has faster access characteristics, thereby mediating between the large capacity of high-latency storage and the speed requirements of access operations.
2Productivity
If more access nodes are added to process requests, then request handling capacity is increased, but system complexity increases
Solution Approach 1:
The system segments the prediction and pre-fetch functionality into individual access nodes, where each node independently maintains its own access history and generates predictions for objects it serves. This segmentation allows the system to scale to multiple access nodes without requiring centralized coordination, thereby managing complexity while increasing request handling capacity.
Solution Approach 2:
Each access node performs self-service by independently maintaining its access history, inverting it to generate predictions, and initiating pre-fetch operations without requiring external coordination. This autonomy simplifies the overall system architecture while enabling scalable deployment across multiple access nodes.
3Measurement precision
If access history is maintained for each access node, then prediction accuracy is improved, but memory usage increases
Solution Approach 1:
The system applies local quality by maintaining access histories that are specific to each access node's local access patterns rather than a global history. Each node's prediction accuracy is optimized for its local context, and the history size can be tuned as a local parameter to balance prediction quality against memory consumption at each node.
4Loss of time
If pre-fetching is performed proactively, then access time is reduced, but computing resource usage increases
Solution Approach 1:
The system performs partial pre-fetching by selecting only a subset of predicted objects for pre-fetch operations rather than fetching all predicted objects. The number of objects to pre-fetch can be configured as a parameter, allowing the system to balance the benefit of reduced access time against the cost of computing resources used for pre-fetching operations.
Data Source
AI summary
Requests to access specific ones of a plurality of stored objects are processed by multiple access nodes. A separate access history is maintained for each access node. Each access history identifies stored objects most recently accessed through the specific node. A separate predicted access future is maintained for each stored object. A predicted access future associated with a specific stored object can be in the form of a listing of stored objects statistically predicted to be those most likely to be accessed within a given temporal proximity after the specific stored object is accessed. Each predicted access future is determined based on inversion of maintained access histories. Responsive to receiving an access request for a specific stored object, the predicted future associated with the requested object is read, a specific number of additional stored objects identified in the associated predicted future is pre-fetched from slower to faster storage.