Distributed Email Threading via Parallel Node Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing email threading technologies face challenges in reconstructing accurate email threads due to issues like missing emails, subject changes, and varying email formats, especially when dealing with large datasets and diverse email clients across different time zones.
Innovation Solution
A parallel scalable distributed algorithm is implemented using a NoSQL database with hierarchical keys, where emails are organized by subject hash and sent date, allowing for efficient retrieval and deduplication of redundant emails across multiple nodes, ensuring accurate thread reconstruction even with incomplete threads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional email threading algorithms are used, then thread reconstruction can be performed, but accuracy deteriorates when dealing with large datasets, missing emails, subject changes, and diverse email formats
Solution Approach 1:
The email threading system is divided into multiple independent nodes that each process specific subsets of emails. Each node independently reconstructs threads for assigned email sets, allowing parallel processing while maintaining accuracy. The segmentation enables the system to handle large datasets without sacrificing thread reconstruction accuracy, as each node focuses on a manageable subset of emails with complete header information.
Solution Approach 2:
A centralized database serves as an intermediary between multiple processing nodes, storing complete email corpus with all header information. The database provides consistent data access to all nodes, ensuring that each node receives complete email data including messages, headers, and contained emails. This intermediary mechanism enables accurate thread reconstruction across distributed nodes while handling diverse email formats and conditions.
2Reliability
If all emails are processed centrally, then complete thread reconstruction is possible, but processing time and computational resources increase significantly with large datasets
Solution Approach 1:
The email corpus is segmented into multiple partitions, with each partition assigned to a separate processing node. Each node independently processes its assigned partition and reconstructs threads for emails within that partition. This segmentation enables parallel processing, dramatically reducing total processing time while maintaining complete thread reconstruction accuracy through coordinated data access via the centralized database.
Solution Approach 2:
Emails are pre-loaded into the centralized database with complete header information, message content, and contained emails before distribution to processing nodes. This preliminary action ensures that all necessary data is available locally at each node, eliminating the need for repeated data fetching during processing and significantly reducing processing time while maintaining reconstruction completeness.
3Reliability
If redundant emails are retained in the database, then complete thread information is preserved, but storage requirements and data processing overhead increase
Solution Approach 1:
The system creates a centralized copy of the complete email corpus with all header information, message content, and contained emails stored in the database. Each processing node accesses this centralized copy rather than storing multiple redundant copies locally. This copying approach preserves complete thread information for accurate reconstruction while avoiding the storage overhead of duplicating entire email corpora at each node, optimizing the balance between information completeness and storage efficiency.
Data Source
AI summary
Systems, apparatuses, methods, and computer readable mediums for implementing a scalable distributed email threading algorithm. A database is created for storing a plurality of emails organized by subjects and relaxed checksums. Each node of a plurality of nodes retrieves a different subject for processing, and each node reconstructs an email discussion thread from a corresponding retrieved subject. A given node may merge incomplete threads which are related but which have different subjects. Then, the nodes may write the reconstructed threads back to the database.