Adaptive Screening for Compressed Data Block Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in storage and retrieval of individually accessible data units due to limitations in compression techniques and indexing methods, which result in high storage requirements and slow access times.
Innovation Solution
The system compresses multiple records into blocks, allowing for efficient compression and indexing, using adaptive screening data structures and hierarchical indexing to quickly locate and retrieve records without the need for full decompression, thereby reducing storage needs and access time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If records are compressed individually, then storage space is reduced, but compression ratio is limited
Solution Approach 1:
The patent combines multiple individual records into a single compressed block, allowing the compression algorithm to work on aggregated data rather than isolated records. This merging approach enables higher compression ratios because the algorithm can identify and eliminate redundancies across record boundaries, while still maintaining the ability to retrieve individual records through selective decompression of the block.
2Quantity of substance
If compression techniques provide high compression ratios, then storage efficiency improves, but the ability to start decompression from arbitrary locations is lost
Solution Approach 1:
The patent segments the compressed data into blocks, where each block contains multiple compressed records. The system maintains metadata that maps individual records to their positions within blocks, enabling the decomposition process to start at specific record locations within a block without requiring decompression from the beginning of the entire compressed file. This segmentation preserves random access capability while maintaining high compression ratios.
3Measurement precision
If an index has an entry for each record, then record location is precise, but index size becomes large
Solution Approach 1:
The patent extracts only the essential indexing information needed for efficient record location, rather than maintaining a complete entry for every single record. The index stores metadata at the block level and uses selective decomposition techniques to locate individual records within blocks, reducing the overall index size while maintaining sufficient precision for record retrieval.
4Loss of time
If screening data structures are used to avoid decompression, then access time is reduced, but false positives increase when checking multiple structures
Solution Approach 1:
The patent implements a dynamic screening mechanism that adaptively manages multiple screening data structures. The system adjusts which screening structures to check based on the query characteristics and maintains false positive rates within acceptable bounds by selectively applying screening at different levels of the data hierarchy. This dynamic approach reduces access time for true positives while controlling the overall false positive rate across multiple screening structures.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Managing data includes: receiving at least one group of individually accessible data units, each data unit identified by a key value, with key values of the received data units being sorted; and processing the data units for storage in a data storage system. The processing includes: storing a plurality of blocks of data; and providing an index that includes an entry for each of the blocks; and generating a plurality of screening data structures associated with the stored blocks for determining a possibility that a data unit that includes a given key value was included in the group of individually accessible data units, including a first screening data structure for screening a first set of one or more of the plurality of blocks and a second screening data structure for screening a second set of one or more of the plurality of blocks.