Stream Analysis Application for Mobile Data Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer systems handling large data streams from mobile communication devices face challenges in identifying and eliminating erroneous data in real-time, which leads to increased storage costs and less accurate analysis due to 'garbage in, garbage out' phenomena.
Innovation Solution
A method involving a message queueing system with a stream analysis application that scores data based on a model developed from historic data, discards or corrects erroneous data, and monitors sources to blacklist or unblacklist them, ensuring only accurate data is processed and stored.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data from multiple sources is stored and processed without filtering, then complete data coverage is achieved, but storage costs increase and data quality deteriorates due to erroneous data
Solution Approach 1:
The system performs preliminary analysis of data streams using trained machine learning models before storing data, identifying and eliminating erroneous data in advance. This preliminary action prevents bad data from entering the storage system, reducing storage volume while maintaining data quality.
Solution Approach 2:
The system extracts and removes erroneous data from the data streams through automated analysis and identification mechanisms. By taking out only the problematic data portions while retaining valid data, the system reduces storage requirements without compromising data quality.
2Reliability
If manual review and filtering of data sources is implemented, then data accuracy improves, but processing time and operational complexity increase
Solution Approach 1:
The system employs automated machine learning models that self-evaluate and classify data quality without human intervention. The models automatically identify erroneous data patterns and eliminate them, achieving high data accuracy while maintaining rapid processing speeds without manual review bottlenecks.
Solution Approach 2:
The system replaces manual data review processes with automated machine learning-based analysis mechanisms. This substitution eliminates the time-consuming nature of human review while maintaining or improving data accuracy through consistent, scalable automated evaluation.
3Reliability
If data filtering and analysis mechanisms are added to the system, then data quality improves, but system complexity increases
Solution Approach 1:
The system implements a universal data analysis framework that handles multiple data sources and error types through a single integrated machine learning model architecture. This multi-functional approach improves data quality while avoiding the complexity of multiple separate filtering systems for different data types.
Solution Approach 2:
The system introduces machine learning models as intermediary components between data collection and storage/processing. These intermediary models automatically filter and quality-assess data, improving overall data quality while managing system complexity by centralizing the filtering function in a single layer.
4Reliability
If real-time data analysis is performed on all incoming data streams, then erroneous data is eliminated promptly, but computational resources and processing time are consumed
Solution Approach 1:
The system applies partial analysis by focusing computational resources on evaluating specific data quality indicators and error patterns rather than analyzing every aspect of each data point. This selective approach eliminates erroneous data effectively while consuming fewer computational resources than complete exhaustive analysis.
Data Source
AI summary
A method of adapting content distribution. The method comprises receiving a stream of data from a plurality of sources by a computer system, storing the stream of data in a message queueing system executed on the computer system, and analyzing the stream of data based on a model of data content by the computer system, where the model is developed automatically based on training the model with historic data. The method further comprises based on the analysis of the stream of data using the model, adapting the information in the stream of data and after adapting the information in the stream of data, processing the data by a processing system executed on the computer system, where the processing comprises storing at least some of the data in a data store.