News Feed De-duplication via Sequence Alignment and Hyperlink Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users face challenges in efficiently finding relevant information due to information overload from multiple sources, with existing search methods being time-consuming and inefficient, especially on mobile devices with limited screen space.
Innovation Solution
An automated system that de-duplicates contextually similar news feed items by preprocessing and normalizing them, calculating resemblance measures through sequence alignment and hyperlink scores, and clustering them to identify representative items, thereby reducing information overload and improving user experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If users manually search through multiple news sources to find relevant information, then they can access comprehensive information from multiple sources, but the time and effort required increases significantly
Solution Approach 1:
The system automatically performs information aggregation, de-duplication, and presentation without requiring user intervention. The automated system collects news from multiple sources, identifies duplicates through similarity comparison, and presents consolidated results, allowing the system to serve itself rather than requiring manual user processing of each news item.
Solution Approach 2:
The system pre-processes and consolidates information from multiple sources before presentation. By performing de-duplication and similarity comparison in advance, the system prepares consolidated news feeds that are ready for immediate display, eliminating the need for users to perform time-consuming manual searches when they access the feed.
2Loss of information
If the system presents all news feed items from multiple sources, then users have access to comprehensive information, but information overload occurs making it difficult to identify relevant content
Solution Approach 1:
The system extracts and removes duplicate information from the news feed collection. By identifying and eliminating redundant news items through similarity comparison, the system retains only representative items while removing excess duplicates, thereby reducing information overload while preserving comprehensive coverage of unique news content.
Solution Approach 2:
The system applies partial de-duplication by removing only the redundant duplicate items while preserving one representative instance of each unique news story. This selective action maintains information coverage by keeping all unique content while removing only the excessive duplicate portions that cause information overload.
3Loss of information
If users review multiple similar news feed items, then they can see information from different sources, but the user experience deteriorates due to redundancy and limited screen space
Solution Approach 1:
The system merges multiple similar news feed items into a single consolidated representation. By combining duplicate or highly similar items from different sources into one representative news item, the system preserves the diversity of information sources while eliminating redundant content that degrades user experience on limited screen spaces.
4Measurement precision
If the system performs detailed similarity comparison of all news feed items, then de-duplication accuracy improves, but processing time and computational resources increase
Solution Approach 1:
The system segments the de-duplication process into distinct stages: initial filtering to identify potential duplicates, detailed similarity comparison for confirmed candidates, and final verification. This segmentation allows the system to apply computationally intensive detailed comparison only to items that pass initial filtering, thereby maintaining high de-duplication accuracy while improving overall processing speed by avoiding unnecessary detailed comparisons of all items.
Data Source
AI summary
The technology disclosed relates to de-duplicating contextually similar news feed items. In particular, it relates to assembling a set of news feed items from a plurality of electronic sources and preprocessing the set to generate normalized news feed items that share common company-name mentions and token occurrences. The normalized news feed items are used to calculate one or more resemblance measures based on a sequence alignment score and/or a hyperlink score. The sequence alignment score determines contextual similarity between news feed item pairs, arranged as sequences, based on a number of matching elements in the news feed item sequences and a number of edit operations, such as insertion, deletion, and substitution, required to match the news feed item sequences. The hyperlink score determines contextual similarity between news feed item pairs by comparing the respective search results retrieved in response to supplying the news feed item pairs to a search engine.