Television Viewership Data Anomaly Detection and Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for collecting and analyzing television viewing data from set-top boxes face issues such as data corruption, duplication, and loss due to varying collection processes, leading to inaccuracies and reliability concerns.
Innovation Solution
A system that uses aggregation and linear regression models to detect and correct aberrations in television viewing data by comparing actual viewer numbers to expected ranges, applying normalization factors, and flagging deviations to ensure accurate and reliable data cleaning and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If television viewing data is collected through multiple nodes and service providers, then the coverage and quantity of data increase, but data corruption, duplication, and loss occur
Solution Approach 1:
The patent segments the data collection process into distinct nodes (service providers, head ends, aggregation points) and applies specific cleaning operations at each segment. This allows targeted correction of data issues at different levels of the hierarchy without affecting the entire dataset, thereby maintaining high quantity while improving reliability through localized purification.
Solution Approach 2:
The system implements feedback mechanisms where expected viewership numbers are compared with actual reported numbers, and correction factors are calculated based on historical data and statistical models. This feedback loop continuously refines the data quality by identifying and correcting aberrations, ensuring that the large volume of collected data maintains high accuracy through iterative purification.
2Adaptability or versatility
If data collection processes vary across different service providers, then adaptability to diverse networks is improved, but data corruption and inconsistencies increase
Solution Approach 1:
The patent applies local quality by customizing data cleaning operations for each service provider or node based on its specific characteristics and error patterns. Different correction factors and statistical models are applied to different segments, allowing the system to maintain adaptability to diverse network structures while improving precision by tailoring purification methods to local data quality issues.
Solution Approach 2:
The system changes parameters such as correction factors, threshold values, and statistical models based on the specific characteristics of each service provider and data segment. This parameter adaptation allows the system to handle varying collection processes across different networks while maintaining consistent data precision through localized parameter optimization.
3Measurement precision
If aggregation is used to clean viewing data, then data accuracy is improved, but processing complexity increases
Solution Approach 1:
The complex aggregation process is segmented into manageable operations performed at different nodes in the data collection hierarchy. Each node performs localized aggregation and cleaning operations independently, reducing the complexity burden on any single system while maintaining overall accuracy through coordinated purification across multiple segments.
Solution Approach 2:
The system employs self-service mechanisms where data is automatically cleaned and corrected using built-in statistical models and correction factors without requiring manual intervention. This automation reduces processing complexity by eliminating manual data cleaning steps while maintaining high accuracy through algorithmic purification methods.
Data Source
AI summary
Various of the disclosed embodiments contemplate television viewing behavior data collected from a plurality of television set top boxes by using aggregation to detect an excess or a deficit in viewership for a group of television set top boxes. In some embodiments, a group of set top boxes can be associated with a particular television service provider, cable television head-end, or data warehouse. Additionally, some embodiments can clean television viewing behavior data by detecting and correcting aberrant viewership in a time series, e.g. based on a weekly or an approximately monthly frequency. In some embodiments, the aberrant viewership can be detected by calculating a minimum expected number of viewers for a day and comparing it to the actual number of households that reported viewers for that day.