Document Deduplication Using Shared Data Groups and Identifiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for managing customer correspondence, such as customer statements, are inefficient due to the presence of duplicate information, which requires significant processing time and storage space, and the use of identifiers for small amounts of duplicate data can be costly and inefficient.
Innovation Solution
A method and system that combine common data elements across multiple documents into a combined group, replace instances of these groups with identifiers, and store the combined groups in a separate data object, allowing for efficient storage and retrieval while minimizing storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If duplicate information is removed and replaced with an identifier, then storage space is saved, but processing time and logic complexity increase
Solution Approach 1:
The patent applies preliminary action by pre-processing documents to identify and group common data elements before replacement. The system performs data element identification, grouping, and threshold evaluation in advance, storing the structured results for efficient retrieval during compression operations, thereby reducing real-time processing time
Solution Approach 2:
The patent segments the document processing into distinct phases: data element identification, grouping common elements, evaluating against minimum size thresholds, and replacement with identifiers. This segmentation allows each step to be optimized independently and enables parallel processing of multiple documents
2Quantity of substance
If small amounts of duplicate information are replaced with identifiers, then storage space is reduced, but the identifier size may exceed the duplicate information size
Solution Approach 1:
The patent applies parameter changes by introducing a minimum size threshold parameter that controls when replacement with identifiers occurs. By adjusting this threshold parameter, the system optimizes the balance between storage savings and identifier overhead, ensuring replacement only happens when genuinely beneficial
Solution Approach 2:
The patent introduces an intermediary evaluation layer that assesses whether duplicate data elements meet the minimum size threshold before replacement. This intermediary mechanism prevents direct replacement of small elements with identifiers, avoiding the scenario where identifiers would be larger than the original data
3Quantity of substance
If all duplicate information across millions of documents is processed, then storage efficiency improves, but processing time becomes prohibitively expensive
Solution Approach 1:
The patent merges common data elements across multiple documents into grouped structures, identifying patterns that recur frequently. By combining and grouping these elements, the system achieves comprehensive compression across millions of documents while reducing redundant processing through pattern recognition
Solution Approach 2:
The system performs preliminary analysis to identify and group common data elements across documents before the actual compression operation. This pre-grouping creates an optimized data structure that enables efficient bulk processing and replacement operations across large document sets
Data Source
AI summary
A method, a processing device, and a computer program product are provided. At least two data elements common between multiple documents are combined into a combined group. Instances of the combined group in the multiple documents are replaced with a corresponding identifier. The combined group is stored in a data object separate from the multiple documents.