Data Segmentation and Clustering for Large Dataset Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data recording systems face challenges in efficiently searching large datasets, such as those exceeding a terabyte, due to the time-consuming nature of serial data reading and the need for manual definition of patterns, leading to difficulties in detecting patterns and analyzing marginal cases.
Innovation Solution
The implementation of a supervised learning capability that uses search seeds selected from data segment clustering to quickly identify and classify patterns of interest within measurement data, allowing for efficient sorting and classification of data segments into similarity clusters, thereby reducing the time required to detect patterns in huge datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If serial reading method is used to search recorded data, then data can be read sequentially, but the time required to search large datasets becomes significant
Solution Approach 1:
The patent segments the large dataset into multiple partitions and processes them in parallel using multiple CPU cores. Each partition is searched simultaneously, dividing the overall search task into smaller concurrent units, thereby reducing total search time while maintaining high productivity.
Solution Approach 2:
The patent performs preliminary actions by pre-processing data into searchable formats, creating indexes and organizing data structures before the actual search operation. This preparation enables faster retrieval and reduces the time required during the search phase without compromising data processing capacity.
2Measurement precision
If mask-test based pattern search is used, then geometric shape patterns can be detected, but false detection occurs due to data fluctuation and overhanging boundaries
Solution Approach 1:
The patent changes the detection parameters by using similarity-based metrics instead of strict geometric boundary matching. It employs configurable similarity thresholds and multiple pattern variants, allowing the system to accommodate data fluctuations and boundary variations while maintaining accurate pattern detection and reducing false positives.
3Measurement precision
If similarity based pattern search is used, then representative data comparison can be performed, but all data must be tested which is time consuming
Solution Approach 1:
The patent segments the data testing process by dividing the dataset into partitions that are processed in parallel. Instead of testing all data sequentially, multiple partitions are evaluated simultaneously using similarity metrics, maintaining accurate pattern matching while significantly reducing overall processing time through concurrent execution.
Solution Approach 2:
The patent applies partial action by testing a representative subset of data points within each partition rather than exhaustively evaluating every single data point. This selective sampling approach maintains sufficient pattern matching accuracy while reducing the total computational burden and processing time.
4Adaptability or versatility
If manual pattern definition is required for each pattern to detect, then specific patterns can be searched, but it is difficult to analyze marginal cases and requires significant human effort
Solution Approach 1:
The patent enables self-service by allowing the system to automatically learn and generate pattern definitions from the data itself. Instead of requiring manual configuration for each pattern, the system performs unsupervised learning to identify inherent patterns and structures, automatically creating detection rules that adapt to the data characteristics and reduce manual configuration complexity.
Solution Approach 2:
The patent changes the approach from fixed manual pattern definitions to dynamic, data-driven pattern parameters. It employs configurable similarity thresholds, multiple pattern variants, and adaptive matching criteria that can be adjusted based on data characteristics, enabling the system to handle marginal cases and diverse patterns without requiring extensive manual configuration for each scenario.
Data Source
AI summary
A method of analyzing measurement data includes sorting EDSs into similarity clusters including a representative RDS; labeling the similarity clusters; sorting the EDSs or RDSs into sub-clusters; selecting one of the EDSs or RDSs in each of the sub-clusters as a search seed representative of the sub-cluster; sorting measurement EDSs from measurement data into measurement clusters including a representative measurement RDS; determining for each search seed a most similar one of the measurement EDSs or RDSs; identifying the determined most similar one of the measurement EDSs or RDSs for each search seed as a classification segment corresponding to the search seed; sorting new measurement EDSs or RDSs into classification clusters with the classification segments; classifying all the new measurement EDSs or RDSs in each classification cluster the same as the corresponding classification segment; and determining whether patterns of interest are in the measurement data stream based on the classification clusters.