Continuous-Variable Tabular Encoder for Manufacturing Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural networks and transformers are ill-suited for processing manufacturing data due to its diverse value types, non-gaussian distribution, extreme outliers, missing or undefined data, and inefficiencies in tokenization, leading to poor generalizability and prediction accuracy.
Innovation Solution
An attention-based neural network architecture that encodes manufacturing data by tokenizing, vectorizing, and embedding continuous variables, while removing undefined values and applying regression-friendly attention blocks with modified regularization techniques to improve prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional neural networks and transformers are used to process manufacturing data, then the model can handle natural language processing tasks effectively, but it performs poorly on manufacturing data due to diverse value types, non-gaussian distribution, extreme outliers, and missing values
Solution Approach 1:
The patent transforms continuous manufacturing variables into discrete token representations through vectorization and embedding. This parameter transformation changes the data distribution from continuous non-gaussian with outliers to discrete categorical tokens, making the data suitable for transformer processing while maintaining information integrity through learned embeddings
Solution Approach 2:
The patent introduces an intermediary encoding layer that sits between the raw manufacturing data and the transformer model. This intermediary layer performs tokenization, vectorization, and embedding operations, acting as a mediator that adapts the data format and distribution to match transformer requirements while preserving the underlying manufacturing information
2Ease of operation
If tokenization is applied to continuous manufacturing variables, then the data can be processed by transformers, but the tokenization process creates inefficiencies and loses information about continuous value distributions
Solution Approach 1:
The patent maps continuous manufacturing variables into a high-dimensional embedding space rather than simple discrete tokens. This dimensional transformation allows the model to represent continuous value distributions through vector embeddings, preserving information about magnitude and relationships while maintaining compatibility with transformer architecture
Solution Approach 2:
The patent segments the tokenization process into multiple stages: first vectorizing continuous variables into fixed-length representations, then embedding these vectors into high-dimensional spaces. This segmented approach allows each stage to handle specific aspects of the transformation, preserving information while enabling transformer processing
3Device complexity
If standard regularization techniques are used in transformers, then the model structure remains simple, but prediction accuracy deteriorates on manufacturing data with outliers and non-gaussian distribution
Solution Approach 1:
The patent modifies regularization parameters and techniques specifically for manufacturing data characteristics. This includes adjusting dropout rates, L2 regularization strengths, and normalization parameters to account for non-gaussian distributions and outliers, improving prediction accuracy without fundamentally changing the transformer architecture
Data Source
AI summary
A systems and methods for implementing attention-based neural networks, attention modules, regularization techniques, and unique data encoding such as for sequential tabular data and/or manufacturing data is provided. The attention-based neural networks may include a high dropout and unique softmax regularization. The encoding may attend to missing or undefined data as well as numerous data types common to manufacturing data.