Modulus Reduction Arrays for Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data deduplication methods struggle with frequent data updates, leading to replication delays and inefficiencies, especially when dealing with large datasets and frequent changes across separate processing windows.
Innovation Solution
The use of modulus reduction arrays, which leverage mathematical properties to break down long arrays into shorter prime-numbered arrays, reducing storage needs and processing time by allowing in-memory handling of large streaming changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data tables with one row per object are used to track each object individually, then data replication can be performed, but the sheer size of the data tables and associated storage speed delays the update process
Solution Approach 1:
The patent divides the traditional single large data table into multiple smaller modulus reduction arrays (MRA). Each MRA tracks a subset of objects, allowing parallel processing and reducing the time complexity of updates. The segmentation enables the system to handle large datasets by distributing the tracking burden across multiple smaller structures rather than one monolithic table.
Solution Approach 2:
The patent creates simplified copies of object state information in the MRA structures. Instead of storing complete object data in traditional tables, the system uses compact representations (object IDs and version numbers) in the modulus reduction arrays. This copying approach reduces storage requirements and speeds up access while maintaining replication accuracy.
2Reliability
If traditional solutions track each object individually with permanent disk delays, then data can be replicated, but the replication process falls behind when objects are updated frequently
Solution Approach 1:
The patent implements dynamic tracking using modulus reduction arrays that can be updated in memory without permanent disk delays. The system dynamically adjusts to frequent updates by using in-memory data structures that support rapid modifications. This dynamic approach allows the replication process to keep pace with frequent object updates, preventing the replication lag that occurs with traditional disk-based solutions.
Solution Approach 2:
The patent changes the fundamental parameters of data storage from traditional disk-based tables to in-memory modulus reduction arrays. This parameter change includes using different data structures (arrays with mathematical properties) and changing the storage medium (memory vs. disk). These parameter changes enable the system to handle frequent updates at much higher speeds while maintaining complete replication.
3Reliability
If multiple syncs are performed for objects updated multiple times in short succession, then all updates are captured, but the default update process requires two or more syncs when only the latest sync would have been sufficient
Solution Approach 1:
The patent performs preliminary actions by pre-processing update information and storing it in the modulus reduction arrays before replication is needed. The system maintains a running record of all updates in the MRA structures, including version numbers and timestamps. This preliminary tracking ensures that when replication occurs, the system can identify the latest update immediately without needing to perform multiple synchronization passes, thus capturing all necessary updates in a single sync operation.
Solution Approach 2:
The patent enables the replication process to skip redundant synchronization operations by using the efficient lookup capabilities of the modulus reduction arrays. The mathematical properties of the MRA structures allow the system to quickly determine which objects have been updated and directly replicate only those objects. This skipping mechanism eliminates unnecessary sync operations, reducing synchronization time while ensuring all updates are captured.
4Adaptability or versatility
If large data tables are used to store millions of rows per account, then all objects can be tracked, but write locks and other delay situations occur
Solution Approach 1:
The patent segments the large data tracking task into multiple smaller modulus reduction arrays. Each MRA handles a portion of the objects, allowing concurrent updates without requiring locks on the entire dataset. This segmentation eliminates write lock bottlenecks that occur in traditional single-table systems, as multiple operations can proceed in parallel across different array segments.
Solution Approach 2:
The patent replaces the mechanical disk-based table system with an in-memory array-based system. This substitution eliminates the physical constraints of disk I/O and write locks that plague traditional systems. The in-memory MRA structures provide lock-free or lock-minimized update operations, dramatically improving the smoothness and speed of update processes while maintaining the ability to track millions of objects.
Data Source
AI summary
A computer-implemented method is disclosed for data processing. The method includes receiving real-time streaming data that includes data object identifiers, arranged in a sequenced source data array, for source data objects. The method also includes determining an integer number N of data object identifiers in the source data array and selecting three mutually prime integers N1, N2, N3 such that N1 is greater than N2, N2 is greater than N3, and an arithmetic product of N1, N2, N3 is greater than N. The method further includes generating a first, second and third modulus reduction arrays of lengths, respectively, N1, N2, and N3. The method also includes initializing the first, second and third modulus reduction arrays with dummy values, storing the source data objects in a source data object array, and storing the first, second and third modulus reduction arrays in a short term memory.