Data Stream Analytics Using Hash-Based Incremental Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing big data analytics systems face challenges in performing real-time analysis of data streams from streaming devices, as they often rely on batch processing, which is inadequate for the constant updates from streaming sources like IoT devices and social media, limiting the ability to provide timely analytics and updates.
Innovation Solution
A dynamic pipeline system utilizing a data stream manager that generates hashes of real-valued vectors, indexes them, and updates data structures in real-time, enabling incremental analysis and machine learning to analyze data streams from multiple sources, including IoT devices and social media, for real-time analytics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If batch processing is used for big data analytics, then system complexity is reduced and processing is simpler, but real-time analysis capability is lost and response time increases
Solution Approach 1:
The system transitions from static batch processing to dynamic stream processing. The data stream manager dynamically processes data objects as they arrive in real-time, maintaining the ability to respond immediately to new data while managing complexity through structured processing pipelines that handle streaming data continuously rather than in fixed batches.
Solution Approach 2:
The patent implements continuous processing of data streams through the data stream manager that operates continuously rather than in periodic batches. The system maintains continuous updates to data structures and continuous analysis as data objects arrive, eliminating the start-stop nature of batch processing and ensuring real-time responsiveness.
2Loss of time
If real-time analysis of data streams is implemented, then response time is improved and timeliness is enhanced, but system complexity increases and processing overhead grows
Solution Approach 1:
The system segments the real-time processing function into a dedicated data stream manager component that is separate from traditional batch processing systems. This segmentation allows real-time processing to be isolated and optimized independently, reducing the complexity burden on the overall system by confining stream processing logic to a specialized module.
Solution Approach 2:
The data stream manager acts as an intermediary layer between data sources and analysis systems. It receives data objects from streaming sources, processes them in real-time, and feeds results to appropriate systems, thereby mediating the complexity of real-time processing and shielding other system components from the intricacies of stream handling.
3Measurement precision
If incremental analysis and updates are performed continuously, then data freshness is improved and analysis accuracy increases, but processing overhead increases and computational resources are consumed continuously
Solution Approach 1:
The system performs partial updates to data structures based on the specific characteristics of each incoming data object rather than complete reprocessing. The data stream manager selectively updates only the necessary portions of data structures when new data objects arrive, avoiding unnecessary computational overhead while maintaining analysis accuracy.
Solution Approach 2:
The patent employs parameter changes in the form of evolving data structure representations that adapt to incoming data. The system modifies parameters such as data object identifiers, timestamps, and analysis results dynamically as new data arrives, allowing efficient incremental updates without requiring complete recomputation of previous analyses.
Data Source
AI summary
Examples disclosed herein involve data stream analytics. In examples herein, a data stream may be analyzed by computing a set of hashes of a real-valued vector, the real-valued vector corresponding to a sample data object of a data stream; generating a list of data objects from a database corresponding to the sample data object based on the set of hashes, the list of data objects ordered based on similarity of the data objects to the sample data object of the data stream; and updating a data structure representative of activity of the sample data object in the data stream based on the list of data objects, the data structure to provide incremental analysis corresponding to the sample data object.