PLS-Tree Data Clustering for High-Dimensional Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data analysis methods, such as regression trees and neural networks, face computational intensity and inefficiency when dealing with large data sets exceeding 10-20 variables, particularly in industries like semiconductor manufacturing and pharmaceutical research, where data sets can have thousands of variables and complex relationships between process and yield variables.
Innovation Solution
The implementation of a partial least squares (PLS) approach for data analysis, which partitions large data sets into smaller, more homogeneous clusters while preserving relationships between process and yield variables, using a top-down method that efficiently handles multiple collinear variables and missing data, and operates on both binary and continuous Y-variables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If regression trees or neural networks are used to analyze large data sets with many variables, then the analysis can capture complex relationships between variables, but the computational cost and processing time increase significantly
Solution Approach 1:
The patent segments the large data set into smaller homogeneous clusters using PLS analysis. By dividing the data into manageable groups based on spectral similarities and process parameters, the computational burden is reduced while maintaining analysis accuracy within each cluster. This segmentation allows regression trees to be applied efficiently to smaller subsets rather than the entire large data set.
Solution Approach 2:
The patent replaces the direct application of computationally intensive neural networks or regression trees on the entire large data set with a two-stage approach: first using PLS (a more efficient dimensionality reduction technique) to cluster and preprocess the data, then applying regression trees only to the clustered subsets. This substitution of the analytical mechanism significantly reduces computational time while preserving the ability to capture complex variable relationships.
2Manufacturing precision
If traditional clustering methods are used to organize large data sets, then data can be grouped into homogeneous subsets, but the methods struggle with data sets having more than 10-20 variables
Solution Approach 1:
The patent introduces PLS analysis as an intermediary technique between the raw high-dimensional data and the traditional clustering methods. PLS transforms the original variables with many collinear relationships into a smaller set of uncorrelated latent variables, making the data suitable for traditional clustering algorithms. This intermediary transformation enables handling of data sets with thousands of variables while maintaining cluster homogeneity.
Solution Approach 2:
The patent changes the parameter space by transforming the original variables into PLS scores and loadings. This parameter transformation reduces the dimensionality of the data while preserving the essential variance and relationships, allowing clustering methods to effectively handle data sets with far more than 10-20 original variables by operating in the transformed parameter space.
3Productivity
If data are partitioned into smaller clusters, then internal variation within clusters is minimized and analysis efficiency improves, but the computational complexity of determining optimal partition points increases
Solution Approach 1:
The patent performs preliminary PLS analysis and spectral preprocessing before the actual clustering partitioning. By pre-processing the data to extract meaningful latent variables and establish baseline homogeneity criteria, the subsequent partitioning algorithm operates more efficiently. This preliminary action reduces the search space and simplifies the determination of optimal partition points, improving overall analysis efficiency without excessive computational complexity.
Data Source
AI summary
A method and system for partitioning (clustering) large amounts of data in a relatively short processing time. The method involves providing a first data matrix and a second data matrix where each of the first and second data matrices includes one or more variables, and a plurality of data points. The method also involves determining a first score from the first data matrix using a partial least squares (PLS) analysis or orthogonal PLS (OPLS) analysis and partitioning the first and second data matrices (e.g., row-wise) into a first group and a second group based on the sorted first score, the variance of the first data matrix, and a variance of the first and second groups relative to the variances of the first and second data matrices.