Geometric Filters and Minimum Spanning Trees for Data Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing repository systems lack an efficient method to determine the similarity between sets of data elements, particularly in high-dimensional spaces, leading to difficulties in managing and searching large volumes of data effectively.
Innovation Solution
The implementation of an approximate minimum spanning tree (MST) and geometric filters to compress data sets, allowing for rapid searching and similarity determination by projecting bit sequences into lower dimensional spaces and calculating edge costs based on symmetric differences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional repository structures are used to store large amounts of data, then data storage capacity is improved, but the ability to determine similarity between data sets deteriorates
Solution Approach 1:
The patent extracts the similarity determination capability from the traditional repository structure by introducing a separate indexing system. The indexing system processes data sets and creates inverted indexes that map content features to document identifiers, enabling similarity search independent of the underlying repository storage structure.
Solution Approach 2:
The patent introduces an intermediary indexing system between the data sets and the search query. The inverted index acts as a mediator that transforms the search problem from direct data comparison to index lookup, significantly improving similarity detection efficiency without changing the repository storage structure.
2Quantity of substance
If data sets are stored in high-dimensional spaces, then data storage capacity is improved, but search efficiency deteriorates
Solution Approach 1:
The patent segments the high-dimensional data into discrete content features and their corresponding occurrences. By breaking down the data into atomic units (words, phrases, or other content elements) and tracking their frequencies and positions, the system transforms complex high-dimensional similarity search into simpler feature-based indexing and matching operations.
Solution Approach 2:
The patent transforms the search problem from high-dimensional continuous space to a lower-dimensional discrete feature space. By representing data sets as collections of discrete content features with associated metadata (frequency, position), the system reduces the dimensionality of the search space while preserving similarity information.
3Measurement precision
If exact minimum spanning tree algorithms are used, then measurement precision is improved, but computational time increases
Solution Approach 1:
The patent applies partial action by computing minimum spanning trees only for subsets of data sets rather than all data sets simultaneously. By processing data sets in batches and building MSTs incrementally, the system achieves sufficient precision for practical purposes while significantly reducing computational time requirements.
Solution Approach 2:
The patent changes the parameters of the MST algorithm by using approximation techniques and heuristic methods instead of exact algorithms. By adjusting parameters such as precision thresholds and computational depth, the system achieves acceptable measurement precision with much lower computational cost.
Data Source
AI summary
Techniques for compressing a data set in a manner such that the data set is represented as a geometric filter are disclosed. Here, the geometric filter has a small memory footprint, thereby enabling the data set to be indexable without consuming a large amount of memory. One bits can be represented within a bit vector portion of the geometric filter or in an index portion of the geometric filter. Techniques are also provided to determine a level of similarity between two data sets, where the data sets have been compressed such that the data sets are representable as geometric filters. The data sets can be indexed without consuming large amounts of memory, and the similarity determination can be performed rapidly. Approximate minimum spanning trees (MSTs) are also generated to find an optimal compression for the data sets, such that the indexes are smaller and searching operations are faster.