Real-Time Streaming Data Categorization via Unsupervised Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI/ML systems are brittle and require retraining when new data types are introduced, struggle with concept drift, and lack mechanisms to detect outliers or noise, forcing new data into existing categories.
Innovation Solution
A real-time data categorization system that uses unsupervised clustering to dynamically create new categories from streaming data without prior knowledge, employing methods like Delaunay triangulation and Parzen Window Density Estimation to identify clusters and classify data points, allowing for the detection of outliers and noise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional AI/ML systems are used for data categorization, then classification accuracy on training data is improved, but the system requires retraining when new data types are introduced and cannot handle concept drift
Solution Approach 1:
The system dynamically adapts to new data types by continuously learning from streaming data without requiring retraining. The categorization system evolves its classification rules in real-time as new concepts emerge, making the system both accurate and adaptable simultaneously
Solution Approach 2:
The system performs preliminary unsupervised clustering on incoming data streams to identify potential new categories before they are formally recognized. This preliminary action allows the system to prepare for concept drift and new data types proactively, maintaining accuracy while gaining adaptability
2Quantity of substance
If traditional AI/ML systems force data into existing categories, then categorization completeness is improved, but the system cannot detect outliers or noise
Solution Approach 1:
The system extracts and isolates outliers and noise from the main data stream by performing unsupervised clustering that naturally separates anomalous data points. These outliers are taken out of the forced categorization process and handled separately, allowing complete categorization of normal data while reliably detecting anomalies
Solution Approach 2:
The system introduces an intermediary unsupervised clustering layer between raw data and final categorization. This intermediary process identifies outliers and noise without preventing complete categorization of valid data, serving as a mediator that protects reliability while maintaining completeness
3Adaptability or versatility
If unsupervised clustering is performed on the entire data stream continuously, then new category detection is improved, but the computational complexity and processing time increase
Solution Approach 1:
The system performs unsupervised clustering periodically or at triggered intervals rather than continuously on the entire data stream. This periodic action maintains new category detection capability while significantly reducing computational complexity and processing time
Solution Approach 2:
The data stream is segmented into manageable chunks for unsupervised clustering analysis. By processing segments rather than the entire stream continuously, the system detects new categories effectively while keeping computational complexity tractable
4Measurement precision
If the system maintains a large pool of unclassified data for clustering analysis, then new category identification accuracy is improved, but the memory requirements and processing delay increase
Solution Approach 1:
The system performs clustering analysis on a partial subset of unclassified data that is sufficient to identify new categories, rather than analyzing the entire pool. This partial action maintains identification accuracy while reducing processing delay and memory requirements
Data Source
AI summary
System and method for real-time data categorization of streaming data output from a data collection system, wherein the categorization system and method has no initial knowledge of a plurality of data categories to which ones of the data in the streaming data can be assigned, each of the plurality of data categories associated with a data cluster. The system, and corresponding methodology, are operative to check each one of the data, as received, against any known data categories and, if one of the data fits one or more of the known data categories, classifying the one of the data according to the one or more of the known data categories, otherwise adding the one of the data to a pool of unclassified data; execute, when the pool of unclassified data reaches a threshold, an unsupervised clustering method on the pool to identify any previously uncategorized clusters of data and define one or more new data categories for any such previously uncategorized clusters; use, if a new data category is defined for a previously uncategorized cluster of data, each of the previously uncategorized clusters to define a shell for which previously unclassified data can be checked for inclusion and assigning any such unclassified data within the shell to the new data category; and, output the categorized data to a data analysis system.