Rollover Strategies in N-bit Dictionary Compressed Column Store
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Dictionary compressed column-oriented databases face inefficiencies and resource consumption issues when dealing with high cardinality columns, as existing memory blocks must be re-encoded or new blocks allocated to accommodate new unique values, leading to delays and data inconsistencies.
Innovation Solution
Implementing a storage rollover system that creates virtual memory blocks within available space to store new tokens using a higher bit encoding without reconfiguring existing data, allowing efficient allocation of memory resources and avoiding wastage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If memory blocks are re-encoded to accommodate new tokens, then storage capacity is improved, but system processing is delayed and data inconsistencies occur
Solution Approach 1:
The patent segments the storage system into multiple memory blocks with different encoding capacities. When a memory block reaches its capacity, new tokens are directed to alternative blocks or higher-capacity blocks rather than re-encoding the entire block, thus avoiding processing delays while maintaining storage expansion capability
Solution Approach 2:
The patent introduces an additional dimension to storage capacity by implementing multiple encoding types (e.g., 8-bit, 16-bit, 32-bit) across different memory blocks. This allows the system to accommodate new tokens by selecting appropriate blocks with higher encoding capacity rather than re-encoding existing blocks, eliminating processing delays
2Quantity of substance
If memory blocks are re-encoded to accommodate new tokens, then storage capacity is improved, but data inconsistencies occur during active access
Solution Approach 1:
By segmenting storage into multiple independent memory blocks with fixed encodings, the patent allows individual blocks to be accessed without affecting others. New tokens are placed in alternative blocks rather than re-encoding active blocks, preventing data inconsistencies during concurrent access
Solution Approach 2:
The system pre-allocates multiple memory blocks with different encoding capacities before they are needed. When storage capacity is exhausted in one block, the system can immediately redirect new tokens to pre-prepared alternative blocks, avoiding the need for re-encoding operations that would cause data inconsistencies
3Reliability
If new memory blocks are allocated instead of re-encoding, then data consistency is maintained, but memory resources are wasted
Solution Approach 1:
The patent creates a universal storage architecture where multiple memory blocks with different encoding capacities serve as a pooled resource. When one block is full, the system can allocate tokens to other blocks in the pool, eliminating waste while maintaining data consistency through uniform access protocols
Solution Approach 2:
The system dynamically changes the parameter of encoding capacity selection based on available space. Instead of wasting memory by always allocating new blocks, the system selects from available blocks with appropriate encoding capacities (8-bit, 16-bit, 32-bit), optimizing resource utilization while maintaining data consistency
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for rollover strategies in an n-bit dictionary compressed column store. An embodiment operates by receiving a new value for addition to a compressed column store. It is determined that a maximum storage capacity for tokens in the compressed column store has been reached for the data dictionary. The compressed column store is converted into a composite store including the existing compressed column store and a newly created flat store. The new value is stored in the flat storage portion of the composite store.