Memory-Efficient Hypercube Cache Versioning With Virtual Tuples
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing query evaluation systems face challenges in efficiently managing large and complex hypercube data with changing data over time, requiring substantial memory resources and causing scalability bottlenecks due to read and write locks, especially when dealing with long-running read sessions.
Innovation Solution
Implementing a multi-versioned concurrency control system that snapshots and versions data at the value object level, using snapshot chains and anchor objects to ensure data consistency without blocking, and efficiently managing memory usage through localized storage and cleanup operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire hypercube is copied to ensure data consistency during updates, then data consistency is maintained, but memory resources are substantially consumed
Solution Approach 1:
The patent segments the hypercube data into individual value objects, each with its own version history. Instead of copying the entire hypercube, only specific value objects that need updates are versioned independently. This segmentation allows the system to maintain data consistency for updated objects while avoiding the memory overhead of full hypercube copies.
Solution Approach 2:
The patent implements selective copying by creating snapshots only for specific value objects that are being updated, rather than copying the entire hypercube. Each value object maintains its own snapshot chain, allowing the system to copy minimal data necessary for consistency while dramatically reducing memory consumption compared to full hypercube replication.
2Reliability
If read and write locks are used to manage concurrent access, then data consistency is ensured, but scalability is bottlenecked especially for long-running read sessions
Solution Approach 1:
The patent implements dynamic versioning where each value object maintains a chain of snapshots with associated validity periods. Readers can dynamically select appropriate snapshots based on their transaction timestamps, allowing long-running read sessions to access historical data without blocking writers. Writers dynamically create new snapshots and update validity periods, enabling concurrent access without traditional locks.
Solution Approach 2:
The patent introduces snapshot chains and validity period metadata as intermediaries between readers and writers. Instead of direct locked access, readers query the anchor object to determine which snapshots are valid for their transaction time, and writers update the anchor object with new snapshot references. This intermediary mechanism eliminates blocking while maintaining consistency.
3Reliability
If full hypercube copies are stored for versioning, then data consistency across time is maintained, but memory usage increases substantially
Solution Approach 1:
The patent segments versioning storage at the value object level rather than at the hypercube level. Each value object maintains its own snapshot chain, allowing the system to store only the specific versions needed for each data element. This segmentation reduces total memory usage compared to storing full hypercube copies for each version point.
Solution Approach 2:
The patent applies versioning with different retention characteristics to different value objects through their individual snapshot chains. Each value object can maintain its own history independently, allowing the system to optimize memory usage by keeping only necessary historical versions locally at each value object rather than maintaining global hypercube-wide version copies.
Data Source
AI summary
A system, method, and device for indexing to a hypercube is provided. The method includes (i) obtaining an index that maps dimensional values to a collection of locations within a hypercube, (ii) determining that one or more tuples of the hypercube is updated along a particular dimension, and (iii) in response to determining that the one or more tuples of hypercube is updated along a particular dimension, creating a new virtual tuple for each updated row and updating the index to store a pointer to the new virtual tuple for a new particular dimension value, the virtual tuple comprising a pointer to a tuple in the index and a validity period for the tuple.