Digest-Based Duplicate Data Reduction in Marketing Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for reducing duplicate data transmission in online marketing systems are inefficient, as they either require impractical storage burdens or only marginally reduce bandwidth usage, due to the large amount of consumer data being sent periodically, often with little change.
Innovation Solution
A system where marketing services providers maintain digests for each server-side account, using a segment table to eliminate duplicate data by representing each field/value pair only once, allowing for field-by-field suppression of unchanged data during updates, thereby reducing bandwidth and storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire consumer record is resent at each periodic update, then all data is transmitted ensuring completeness, but bandwidth is wasted transmitting duplicate unchanged fields
Solution Approach 1:
The patent extracts only the changed fields from the consumer record and transmits them separately from the unchanged fields. This is achieved by comparing the current record with the previous version, identifying differences, and transmitting only those differences along with a reference to the unchanged portion, thereby eliminating duplicate data transmission while maintaining completeness
Solution Approach 2:
The consumer record is segmented into changed and unchanged fields. The patent divides the data transmission into two parts: a reference to the unchanged fields (which can be stored at the destination) and only the changed fields that need updating. This segmentation allows selective transmission of only necessary data portions
2Loss of energy
If a hash table is used to track sent data, then duplicate records are eliminated, but any field change causes entire record resending
Solution Approach 1:
Instead of treating the entire consumer record as a single unit for duplicate detection, the patent segments the record into individual fields. Each field is tracked separately, allowing the system to identify and transmit only the specific fields that have changed rather than the entire record, thus improving data transmission efficiency
Solution Approach 2:
The patent applies different handling strategies to different fields within the same record based on their change status. Changed fields are transmitted with full detail, while unchanged fields are referenced or omitted entirely. This local differentiation optimizes bandwidth usage without losing necessary information
3Reliability
If the entire consumer record is stored to eliminate duplicates, then duplicate detection is accurate, but storage burden becomes enormous
Solution Approach 1:
The patent segments the stored data into only the essential unique identifiers and change indicators rather than storing complete consumer records. By storing only field-level change information and using incremental updates, the storage burden is dramatically reduced while maintaining accurate duplicate detection capability
Solution Approach 2:
Instead of storing complete consumer records, the patent uses references or pointers to the original records that remain at the source or are previously transmitted. The destination stores only incremental change data or metadata that references the full record, significantly reducing storage requirements while maintaining data integrity
Data Source
AI summary
A machine and method of reducing duplicate transmission data employs one more more digests to track field/value pairs that have previously been distributed. Each digest contains a record table and a segment table. The record table includes anonymous identifier records, each of which contain an anonymous identifier and one or more indexes into the segment table. The segment table comprises an array of every existing data field/value pair. Before distribution of update data, each record is matched to an anonymous identifier record in the record table. The segment values in the prospective distribution record are compared to the digest's anonymous identifier record, used to determine which data has already been distributed, and thus will be suppressed in the distribution.