Cross-Data-Set String Replacement for Storage Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing demand for data storage exceeds available capacity, leading to high costs and the need to choose between purchasing more storage or reducing data retention periods, as existing technologies fail to efficiently compress data sets without significant loss of information.
Innovation Solution
A computing device identifies common strings in multiple data sets, replaces their occurrences with references, and iteratively repeats this process to achieve a predetermined size reduction, using weighted scores based on frequency and length to select strings for replacement, thereby compressing data sets by at least 30%.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If additional storage space is purchased to accommodate increased data amounts, then storage capacity is improved, but storage cost increases
Solution Approach 1:
The patent merges multiple data sets into a single compressed data set by identifying and replacing common strings with references. This consolidation reduces the total storage volume required while maintaining access to all original data, thereby reducing storage costs without sacrificing capacity.
Solution Approach 2:
Instead of storing redundant copies of common strings in each data set, the patent creates a single reference copy and replaces all occurrences with pointers to this reference. This eliminates redundant storage while preserving data integrity and accessibility.
2Duration of action of stationary object
If data retention period is extended to maintain historical data, then information availability is improved, but storage resource consumption increases
Solution Approach 1:
The patent merges multiple historical data sets into a compressed format, allowing extended data retention periods without proportionally increasing storage space. By consolidating redundant information across time periods, the system can retain more historical data within the same storage constraints.
3Volume of stationary object
If data compression is applied to reduce storage size, then storage space is reduced, but data processing complexity increases
Solution Approach 1:
The patent performs preliminary compression during data ingestion or batch processing, establishing the compressed format and reference tables in advance. This upfront effort reduces the complexity of ongoing operations, as the compressed structure remains stable and requires only simple reference lookups during retrieval.
Data Source
AI summary
A computing device may select a plurality of data sets, determine a set of strings that are included in at least two data sets of the plurality of data sets, and select a particular string of the set of strings. The computing device may replace each occurrence of the particular string in the plurality of data sets to create a modified plurality of data sets such that the modified plurality of data sets is smaller in size than the plurality of data sets. The computing device may assign a reference to the particular string and replace each occurrence of the particular string in the plurality of data sets to create a plurality of modified data sets. The computing device may replace may store the reference and the particular string in a table.