Dictionary-Based Data Compression Search Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database search mechanisms for compressed data are inefficient, requiring full dataset scans and consuming significant time due to complexities in storage and reconstruction of compressed data, especially when dealing with large volumes of redundant data.
Innovation Solution
Implementing a dictionary-based compression method that stores data as unique value identifiers with associated positional addresses, allowing for optimized search by determining the range of positional addresses for requested data, reducing the number of scans and improving search efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire dataset is scanned to search for data, then all data can be searched, but the search time and computational overhead increase significantly
Solution Approach 1:
The patent segments the dataset into multiple partitions or blocks, each with associated metadata indicating the presence and location of specific data values. Instead of scanning the entire dataset, the search process divides the data space into manageable segments and only examines relevant partitions, significantly reducing search time while maintaining complete search coverage.
Solution Approach 2:
The patent performs preliminary organization of data into structured partitions with pre-computed metadata that indicates which partitions contain which data values. This preliminary action enables the search process to skip irrelevant partitions directly, avoiding the need to scan entire datasets during actual search operations.
2Quantity of substance
If compressed data storage is used to improve storage efficiency, then storage space is reduced, but data reconstruction and sorting operations become more complex
Solution Approach 1:
The patent extracts only the essential metadata needed for search operations (such as partition boundaries, data value ranges, and location information) from the compressed data structure. This extracted metadata is stored separately and used to guide search operations without requiring full data reconstruction, thereby reducing complexity while maintaining storage efficiency.
Solution Approach 2:
The patent introduces metadata as an intermediary layer between the compressed data storage and the search operation. This metadata acts as a mediator that provides search guidance without requiring the actual data to be decompressed or reconstructed, simplifying the search process while preserving the space benefits of compressed storage.
3Quantity of substance
If data is stored in compressed form with unique value identifiers, then storage efficiency improves, but search operations require additional processing overhead
Solution Approach 1:
The patent performs preliminary organization of compressed data into value-based partitions with pre-computed metadata that directly maps unique value identifiers to their storage locations. This preliminary action eliminates the need for complex processing during search operations, as the metadata directly provides location information for any given value identifier.
Solution Approach 2:
The patent changes the organizational parameter of compressed data from traditional row-based or column-based structures to value-based partitions. This parameter change allows the system to directly locate data by value identifier without requiring complex decoding or reconstruction operations, reducing search processing overhead while maintaining compressed storage efficiency.
Data Source
AI summary
Methods and system are disclosed that relate to optimizing search for data. In one aspect, an attribute vector may include unique value identifiers and be associated with a dictionary structure. For a unique value identifiers stored in an attribute vector and associated with the dictionary structure, start address and end address associated with the unique value identifiers is computed. Based on the computation, a range of positional addresses associated with the unique value identifiers may be generated and stored in a data structure. Upon receiving a request to search for data, the range of positional addresses in which the unique value identifiers may be searched is determined. Based on the determination, a database search engine optimizes the search for data in the attribute vector.