Indexing Compressed Data Blocks for Key-Based Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in storing and retrieving individually accessible data units, particularly in managing compressed data records, as they require decompression from the beginning of a file to access a specific record, which is time-consuming and inefficient.
Innovation Solution
The method involves generating blocks of data by combining multiple data units, compressing them, and creating an index that allows for the location of specific blocks based on key values, enabling efficient access without full decompression, using techniques like sorting and indexing to manage and retrieve records effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data units are compressed into a continuous file for storage efficiency, then storage space is reduced, but access time increases because decompression must start from the beginning of the file
Solution Approach 1:
The patent divides the compressed data file into multiple blocks, where each block contains a specific range of data units identified by key values. This segmentation allows the system to access only the relevant block containing the desired data unit rather than decompressing the entire file from the beginning, thereby reducing access time while maintaining compression efficiency for storage space reduction.
Solution Approach 2:
The patent pre-compresses data units and organizes them into blocks with associated metadata (such as key value ranges) before storage. This preliminary organization enables the indexing system to quickly locate the specific block containing the desired data unit without requiring sequential decomposition of the entire compressed file, thus resolving the contradiction between storage efficiency and access speed.
2Loss of time
If individual data units are stored separately for fast access, then access time is reduced, but storage space increases due to loss of compression efficiency
Solution Approach 1:
The patent segments compressed data into blocks that balance compression efficiency with access speed. Each block contains multiple compressed data units organized by key value ranges, allowing the system to access a small subset of compressed blocks rather than individual uncompressed units. This segmentation maintains compression efficiency for storage while enabling faster access compared to continuous file decompression.
Solution Approach 2:
The patent introduces an indexing structure as an intermediary between the compressed blocks and the access request. The index contains metadata about each block (such as key value ranges), enabling the system to quickly identify and access only the relevant compressed blocks without decompressing unrelated data. This intermediary layer resolves the contradiction by enabling efficient navigation through compressed data.
3Quantity of substance
If compression is applied to reduce storage requirements, then storage space is reduced, but the complexity of data retrieval increases due to decompression requirements
Solution Approach 1:
The patent performs preliminary compression and organization of data units into blocks with associated metadata before storage. This pre-processing creates a structured compressed format where blocks can be independently accessed and decompressed based on their key value ranges, simplifying the retrieval process compared to decompressing entire continuous compressed files.
Solution Approach 2:
The patent segments compressed data into independent blocks that can be individually accessed and decompressed. This segmentation reduces retrieval complexity by limiting the scope of decompression operations to only the necessary blocks identified through the indexing system, rather than requiring decompression of the entire compressed data set.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A method for managing data includes receiving individually accessible data units, each identified by a key value; storing a plurality of blocks of data, each of at least some of the blocks being generated by combining a plurality of the data units; and providing an index that includes an entry for each of the blocks. One or more of the entries enable location, based on a provided key value, of a block that includes data units corresponding to a range of key values that includes the provided key value.