Pub/Sub Message ID Tracking for Data Record Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The publish/subscribe (Pub/Sub) messaging service does not keep track of delivered messages, leading to unnecessary re-sending of already sent messages, which consumes system resources and causes data duplication, complicating data processing services.
Innovation Solution
A system that uses message identifiers associated with data records to track duplication status, storing metadata with file paths and row numbers, and generates a deletion vector to manage duplication, minimizing resource usage and reducing operation load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the Pub/Sub messaging service is used to access remote data sources, then data access capability is improved, but data duplication and resource consumption increase
Solution Approach 1:
The system performs preliminary action by storing message identifiers along with data records when they are first received via Pub/Sub. This preliminary tracking enables subsequent deduplication operations to identify and filter out duplicate messages, resolving the contradiction between enabling flexible data access and preventing data duplication.
Solution Approach 2:
The system implements feedback mechanisms by maintaining a record of processed message identifiers and using this feedback information to determine whether incoming messages are duplicates. This feedback loop allows the system to continue using Pub/Sub for data access while actively preventing redundant data storage and processing.
2Adaptability or versatility
If the Pub/Sub messaging service is used to access remote data sources, then data access capability is improved, but system resource consumption increases
Solution Approach 1:
The system extracts only the necessary information (message identifiers) from the Pub/Sub messages for tracking purposes, while the actual data records are stored separately. This extraction approach enables efficient deduplication without requiring the system to process or store every detail of every message, thereby reducing overall resource consumption while maintaining data access capability.
Solution Approach 2:
The system applies partial action by implementing deduplication logic that processes only the metadata (message identifiers) rather than the entire data payload. This partial processing approach reduces the computational and storage resources required for deduplication operations while still effectively preventing redundant data processing.
3Adaptability or versatility
If the Pub/Sub messaging service is used to access remote data sources, then data access capability is improved, but operational complexity increases
Solution Approach 1:
The system merges the Pub/Sub messaging mechanism with a deduplication tracking mechanism into a unified approach. By combining the data retrieval function with the deduplication tracking function in a coordinated system, it maintains the simplicity and flexibility of Pub/Sub data access while adding deduplication capability without proportionally increasing operational complexity.
Data Source
AI summary
A device for using message identifiers for Publish/subscribe messaging deduplication is described. The system may fetch one or more sets of data records from a data source, and each data record is associated with a message identifier. The system may store the one or more sets of data records in a data file, which is associated with a metadata comprising the message identifier, a file path and a row number for each data record. The system may determine whether one or more of the data records are duplicated based on the associated message identifiers. In response to determining that the one or more data records are duplicated, the system may generate a second metadata comprising the file paths and row numbers associated with the duplicated data records.