Incremental Clustering of Continuous Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data clustering techniques struggle to maintain data continuity when processing continuous data streams, leading to inaccurate analyses and increased reprocessing and storage costs due to the loss of data continuity over time intervals.
Innovation Solution
A method that dynamically adjusts a minor cluster period based on similarity thresholds to preserve data continuity, allowing for incremental clustering and removal of noncritical data, thereby improving clustering performance and reducing storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If clustering is performed on continuous data streams using traditional clustering techniques, then data can be organized into groups based on similarity, but data continuity is lost over time intervals leading to inaccurate analyses
Solution Approach 1:
The patent segments the clustering process into incremental steps by introducing minor cluster periods within major cluster periods. Instead of clustering all data at once, the system divides the data stream into manageable time segments, processes them incrementally, and maintains continuity through overlapping periods. This segmentation allows the system to preserve data continuity while performing accurate clustering analysis.
Solution Approach 2:
The patent implements dynamic adjustment of cluster periods based on data characteristics. The minor cluster period is dynamically determined based on the detected period of the data stream, allowing the system to adapt to varying data patterns. This dynamic approach ensures that clustering maintains accuracy while preserving continuity across different data conditions.
2Productivity
If traditional clustering techniques are used on large continuous data streams, then clustering can be performed, but reprocessing and storage costs increase
Solution Approach 1:
The patent performs preliminary clustering actions by creating minor cluster periods that prepare data for subsequent major cluster period processing. The system pre-processes data in smaller increments, maintaining cluster information that can be directly used in the next processing cycle. This preliminary action eliminates the need for complete reprocessing and reduces storage requirements by retaining only essential cluster metadata.
Solution Approach 2:
The patent discards non-critical data and intermediate processing results that are not needed for maintaining cluster continuity, while recovering and retaining essential cluster information. By selectively discarding redundant data and recovering only the necessary cluster characteristics, the system reduces storage costs and minimizes reprocessing requirements while maintaining clustering efficiency.
Data Source
AI summary
In an approach to incremental clustering of continuous flow data streams to preserve data continuity, one or more computer processors receive a definition of a first major cluster period and a definition of a first minor cluster period. One or more computer processors receive a continuous stream of data. One or more computer processors cluster the continuous stream of data, based on the definition of the first major cluster period and on the definition of the first minor cluster period. One or more computer processors determine a similarity between a first data cluster and a second data cluster, wherein the second data cluster occurs after the first minor cluster period. One or more computer processors determine whether the similarity meets a pre-defined minimum threshold of similarity. One or more computer processors increase the first minor cluster period to a second minor cluster period.