ML-Based Historical Data Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high cost and latency issues associated with storing and retrieving large amounts of historical data, known as 'big data', pose challenges in maintaining efficient storage systems, leading to increased time to market and resource expenditure.
Innovation Solution
A machine learning-based system that determines optimal storage configurations by training models using performance metrics to balance query performance and storage space, employing techniques like genetic algorithms and neural networks to generate and manage snapshots and journals efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large amounts of historical data are stored to meet query demands, then data availability is improved, but storage cost increases
Solution Approach 1:
The patent segments historical data into multiple representations (snapshots and journals) with different storage characteristics. Snapshots store complete historical states at specific points in time, while journals store incremental changes. This segmentation allows the system to store only necessary data portions, reducing overall storage requirements while maintaining data availability for queries.
Solution Approach 2:
The system dynamically adjusts storage parameters based on data access patterns and performance metrics. By changing parameters such as snapshot frequency, journal retention periods, and storage tier allocations, the system optimizes the balance between storage cost and data availability, storing less expensive compressed or archived representations for less frequently accessed historical data.
2Speed
If sufficient storage space is purchased to maintain low latency, then query performance is improved, but storage cost increases
Solution Approach 1:
The patent applies different storage qualities to different data representations based on their access patterns. Frequently accessed recent data is stored in high-performance storage with low latency characteristics, while older or less frequently accessed historical data is stored in lower-cost storage tiers. This local quality differentiation maintains query performance for critical data while reducing overall storage cost.
3Quantity of substance
If data storage capacity is reduced to lower costs, then storage cost decreases, but query latency increases
Solution Approach 1:
The system performs preliminary actions by pre-computing and storing data in multiple representations (snapshots at different time points, journals with varying retention periods) before queries are executed. This preliminary structuring of data allows the query system to quickly locate and retrieve required information without scanning entire storage capacities, maintaining low latency even with reduced storage.
Solution Approach 2:
The patent introduces an intermediary data structure layer between the physical storage and the query interface. The snapshot and journal representations act as intermediaries that organize and index historical data, enabling efficient retrieval paths. This intermediary layer allows the system to serve queries quickly from a reduced storage capacity by directing queries to the most relevant stored representations rather than searching through all stored data.
4Reliability
If more storage space is purchased to maintain performance, then query performance is maintained, but time to market increases
Solution Approach 1:
The system dynamically adapts its storage configuration based on changing data access patterns and performance requirements. Rather than purchasing fixed storage capacity in advance, the system continuously monitors query performance and automatically adjusts storage allocations, snapshot frequencies, and journal retention policies. This dynamic approach allows the system to maintain query performance while minimizing storage costs and accelerating time to market by avoiding over-provisioning.
Data Source
AI summary
Techniques are disclosed relating to using machine learning techniques to predict storage configurations for historical data. In some embodiments, a computer system stores representations of historical data according to a current set of storage parameters. The representations may include snapshots of historical data in a data repository at different points in time. The computer system may receive queries for historical data specifying points in time from which to retrieve the historical data. In some embodiments, the computer system responds to the queries using the stored representations and determines performance metrics for the responses. In some embodiments, the computer system trains a machine learning model using the performance metrics. Based on output of the trained model, the computer system updates the current set of storage parameters. The updating may affect subsequent storage of representations in the data repository, which may advantageously improve query response times and decrease repository storage size.