N-bit Compressed Column Data Array for In-Memory Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing columnar in-memory data stores face challenges in efficiently managing concurrent reader and writer access while maintaining storage efficiency and supporting large data volumes, particularly in handling index vectors with growing value identifiers that exceed n-bit encoding capacity.
Innovation Solution
The solution involves generating a second backing array in a different memory chunk when the first array reaches capacity, allowing concurrent access through semaphores and exclusion mechanisms to prevent data corruption, and ensuring only one writer can perform structural changes at a time, enabling efficient memory representation and performance for delta index vectors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a single backing array is used for index vector storage, then storage efficiency is improved through n-bit compression, but the array cannot accommodate growing value identifiers that exceed capacity
Solution Approach 1:
The patent divides the index vector storage into multiple backing arrays (first backing array, second backing array) instead of using a single large array. Each backing array has a fixed capacity and stores a portion of the index vector. When the first backing array reaches capacity, a second backing array is created to store additional entries, allowing the system to accommodate growing value identifiers while maintaining efficient n-bit compression within each array segment.
2Productivity
If concurrent reader and writer access is allowed, then system throughput is improved, but data corruption can occur during structural changes
Solution Approach 1:
The patent introduces a semaphore as an intermediary synchronization mechanism to coordinate concurrent access. The semaphore acts as a mediator that grants exclusive access to writers during structural changes (such as creating new backing arrays) while allowing readers to continue accessing existing data. This intermediary mechanism prevents data corruption by ensuring that structural modifications are atomic and visible to readers only after completion, thereby maintaining data integrity while supporting high concurrency.
3Speed
If multiple writers can perform structural changes simultaneously, then system responsiveness is improved, but data corruption and inconsistency increase
Solution Approach 1:
The patent implements preliminary action by requiring writers to acquire a semaphore before performing any structural changes to backing arrays. This preliminary acquisition of the exclusion mechanism ensures that only one writer can modify the data structure at a time, preventing concurrent structural changes that would lead to data inconsistency. The semaphore is acquired before the change and released after completion, ensuring atomicity and consistency of all structural modifications.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
As part of a columnar in-memory database, value identifiers are inserted into a backing array in-memory until such time that it is determined that such backing array does not have sufficient capacity. A new backing array is then generated that includes the value identifiers in the old backing array and which has sufficient capacity. The old backing array can be flushed from memory when there are no active operations using such backing array. Such an arrangement allows for readers and non-structural writers to operate concurrently. Related apparatus, systems, techniques and articles are also described.