Screening Data Structures for Compressed Storage Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in storing and retrieving individually accessible data units, particularly in managing compressed data records, where accessing specific records requires decompression from the beginning of a file, leading to high decompression costs and storage inefficiencies.
Innovation Solution
The method involves generating and using screening data structures and indices to efficiently locate and retrieve data units by sorting and compressing data units into blocks, allowing for the determination of whether a data unit is present without full decompression, and using a hierarchical indexing system to reduce storage and access times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data units are compressed to reduce storage requirements, then storage efficiency is improved, but access time increases because full decompression is required to retrieve specific records
Solution Approach 1:
The patent divides the compressed data into blocks, each associated with a screening data structure that can independently determine presence of data units without decompressing the entire file. This segmentation allows selective access to blocks rather than decompressing all data, resolving the contradiction between compression benefits and access speed.
Solution Approach 2:
The patent introduces screening data structures as intermediary components between the compressed data blocks and the retrieval process. These screening structures act as mediators that enable quick determination of data unit presence without full decompression, allowing the system to maintain compression efficiency while enabling fast access through the intermediary screening mechanism.
2Productivity
If screening data structures are generated for each block to enable efficient searching, then search performance is improved, but storage requirements increase
Solution Approach 1:
The patent applies partial action by generating screening data structures only for blocks that contain data units, rather than for all blocks. The system dynamically determines which blocks need screening structures based on actual data presence, avoiding unnecessary storage overhead while maintaining search performance for relevant blocks.
Solution Approach 2:
The patent changes the parameter of storage requirements by using compact bit-level screening data structures that can be efficiently stored. The screening structures use minimal bits per block, and the system adjusts the level of compression and screening based on the actual data characteristics and access patterns, optimizing the balance between storage and search performance.
3Productivity
If data units are sorted by key value before compression, then indexing efficiency is improved, but processing time increases during data ingestion
Solution Approach 1:
The patent applies preliminary action by sorting data units by key value before compression and block formation. This pre-sorting enables efficient indexing and searching without requiring additional sorting operations during retrieval, as the data is already organized in the optimal order for indexing. The preliminary sorting is performed during the initial data ingestion phase.
Solution Approach 2:
The patent maintains continuity of useful action by keeping data units sorted by key value throughout the storage and retrieval process. The sorting order is preserved during compression and block formation, allowing the system to continuously maintain indexing efficiency without interruption. This continuous sorted organization enables fast binary search operations on the index without requiring re-sorting.
Data Source
AI summary
Managing data by: receiving a group of individually accessible data units, each data unit identified by a key value, with key values determined such that the key value identifying a first data unit received before a second data unit occurs earlier in a sort order than the key value identifying the second data unit; and processing the data units for storage in a data storage system. The processing includes: storing blocks of data, the blocks being generated by combining a plurality of the data units; providing an index with entries that enable location, based on a provided key value, of a block that includes a data unit corresponding to the provided key value; and generating one or more screening data structures associated with the blocks for determining, based on a given key value, whether to search the stored blocks for a data unit corresponding to the given key value.