Manufacturing Data Transformer With Missing-Value Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural networks and transformers struggle with manufacturing data due to its diverse value types, non-gaussian distribution, extreme outliers, missing data, and undefined values, which affect regression-based tasks and require improved regularization techniques.
Innovation Solution
An attention-based neural network architecture with an encoder and transformer that vectorizes and embeds tabular data, applies reduction tensors to handle missing values, and uses regression-friendly self- and cross-attention layers with modified softmax and dropout functions for improved regularization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional neural networks and transformers are used for manufacturing data, then general data processing capability is provided, but precision and reliability deteriorate due to diverse value types, non-gaussian distribution, extreme outliers, missing data, and undefined values
Solution Approach 1:
The patent modifies key parameters of the transformer architecture including: (1) replacing standard softmax with modified softmax that handles non-gaussian distribution, (2) adjusting dropout rates to greater than 0.1 for regression tasks, (3) changing normalization approaches to handle diverse value types, and (4) modifying loss functions for regression-friendly training. These parameter changes enable the model to achieve both high precision and reliability on manufacturing data with outliers and missing values.
2Stability of the object's composition
If standard regularization techniques are applied, then model stability is enhanced and training is hastened, but performance deteriorates for regression-based tasks on manufacturing data
Solution Approach 1:
The patent implements dynamic regularization strategies where dropout rates are adjusted to greater than 0.1 specifically for regression tasks, and modified softmax functions dynamically adapt to the non-gaussian distribution of manufacturing data. The model dynamically balances between stability (through controlled regularization) and precision (through task-specific parameter adjustment), allowing it to maintain training stability while achieving superior regression performance.
3Adaptability or versatility
If conventional transformer architecture is used, then general-purpose data processing is achieved, but adaptability to manufacturing data characteristics deteriorates
Solution Approach 1:
The patent applies local quality modifications to specific components of the transformer architecture: modified softmax functions are applied only where needed for non-gaussian data, high dropout rates are applied specifically to attention layers for regression tasks, and data-specific embeddings are used for manufacturing features. This localized adaptation achieves high adaptability to manufacturing data without requiring complete architectural redesign, thus managing complexity effectively.
4Productivity
If missing values are encoded with sparse representation, then sequence length is decreased, but information loss may occur
Solution Approach 1:
The patent uses sparse representation as an intermediary mechanism that efficiently encodes missing values without complete information loss. The sparse embeddings act as mediators that preserve the semantic meaning of missing data while reducing sequence length. This allows the model to maintain processing efficiency while retaining crucial information about missing values through specialized embedding vectors that capture the nature of missingness.
Data Source
AI summary
A systems and methods for implementing attention-based neural networks, attention modules, regularization techniques, and unique data encoding such as for sequential tabular data and/or manufacturing data is provided. The attention-based neural networks may include a high dropout and unique softmax regularization. The encoding may attend to missing or undefined data as well as numerous data types common to manufacturing data.