Hypercube Cache Snapshot Chains for Memory-Efficient Concurrency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing query evaluation systems face challenges in efficiently managing large and complex hypercubes due to the resource-intensive cloning operations required for storing and maintaining data consistency, leading to scalability bottlenecks and long latency spikes.
Innovation Solution
Implementing a multi-versioned concurrency control system that snapshots and versions data at the value object level, using snapshot chains with validity periods, and anchor objects to manage data changes without blocking reading and writing processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire hypercube is copied to ensure data consistency during updates, then data consistency is maintained, but memory resources are substantially consumed
Solution Approach 1:
The patent segments the hypercube into individual value objects that can be independently versioned. Instead of copying the entire hypercube, only specific value objects that have changed are copied and versioned. This segmentation allows the system to maintain data consistency for updated portions while avoiding the memory overhead of copying the complete hypercube structure.
Solution Approach 2:
The patent implements selective copying through snapshotting of individual value objects. When a value object is updated, a snapshot (copy) is created and stored in a snapshot chain, while the original value object remains accessible to reading processes. This copying mechanism ensures data consistency for specific objects without requiring full hypercube duplication.
2Reliability
If reading processes wait for writing processes to complete to ensure data consistency, then data consistency is maintained, but processing bottlenecks increase
Solution Approach 1:
The patent extracts the data consistency mechanism from the traditional locking approach by implementing snapshot-based versioning. Instead of having reading processes wait for writing processes to complete (blocking), the system extracts the updated data into snapshots that are immediately accessible to readers. This allows reading and writing processes to proceed concurrently without blocking each other, maintaining data consistency while improving processing efficiency.
Solution Approach 2:
The patent introduces snapshots as an intermediary between writing processes and reading processes. When a value object is updated, a snapshot is created as an intermediary representation that reading processes can access immediately. This intermediary mechanism allows writers to update data while readers simultaneously access consistent historical versions, eliminating the need for readers to wait for writers to complete.
3Adaptability or versatility
If multiple snapshots are retained to support historical queries, then query flexibility is improved, but storage requirements increase
Solution Approach 1:
The patent implements dynamic snapshot management where snapshots are retained based on active query requirements. The system dynamically determines which snapshots need to be kept in memory versus which can be removed, allowing the snapshot chain to adapt to current query workloads. This dynamic approach maintains query flexibility for historical analysis while optimizing storage by removing unnecessary snapshots that are no longer needed by active processes.
Data Source
AI summary
A system, method, and device for managing updates to a hypercube is provided. The method includes (i) obtaining a hypercube comprising a plurality of locations, wherein a particular location has an anchor object that stores version information for the particular location, and the particular location comprises a first formula that is evaluated to return a value for the particular location, (ii) in response to receiving the request for the value, determining that the particular location comprises the first formula, (iii) in response to determining that the particular location comprises the first formula, evaluating the first formula to obtain a result, and (iv) storing the result for the particular location in association with the version information for the particular location.