Time2Vec Embedding Layer for Machine Learning Temporal Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning architectures struggle to effectively utilize time as a feature, leading to issues such as exploding and vanishing gradients, and require hand-crafted time features which are costly and require domain expertise.
Innovation Solution
The introduction of a new data structure representation called Time2Vec, which decomposes time into a learnable vector representation, allowing for improved exploitation of temporal information and adaptation to periodicity in data without the need for explicit periodicity definition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional time-based inputs are used in machine learning architectures, then the model can process temporal data, but the model suffers from exploding and vanishing gradients and requires hand-crafted time features
Solution Approach 1:
The patent extracts the time feature from conventional inputs and represents it separately using a vector embedding layer. This extracted time representation is then concatenated with other features, allowing the model to handle temporal information without the complexity of hand-crafted features while maintaining training stability through learned embeddings rather than raw time values
Solution Approach 2:
The patent introduces a vector embedding layer as an intermediary between raw time inputs and the model processing. This intermediary transforms time into learned vector representations that are more suitable for neural network processing, eliminating the need for hand-crafted features and improving gradient flow stability
2Adaptability or versatility
If hand-crafted time features are used, then domain expertise can be incorporated, but the feature engineering process is costly and requires significant domain expertise
Solution Approach 1:
The patent enables the model to automatically learn time feature representations through the vector embedding layer during training. Instead of requiring external domain expertise to craft features, the model self-learns optimal time representations from the data, significantly reducing feature engineering time while maintaining adaptability to domain-specific temporal patterns
3Measurement precision
If more training epochs are used to improve accuracy, then model performance increases, but computational resource requirements increase
Solution Approach 1:
The patent applies preliminary transformations to time inputs using vector embeddings before they enter the main model processing. This preprocessing step creates more informative features that accelerate convergence, allowing the model to achieve high accuracy with fewer training epochs and thus reducing overall computational resource consumption
Data Source
AI summary
Described in various embodiments herein is a technical solution directed to decomposition of time as an input for machine learning, and various related mechanisms and data structures. In particular, specific machines, computer-readable media, computer processes, and methods are described that are utilized to improve machine learning outcomes, including, improving accuracy, convergence speed (e.g., reduced epochs for training), and reduced overall computational resource requirements. A vector representation of continuous time containing a periodic function with frequency and phase-shift learnable parameters is used to decompose time into output dimensions for improved tracking of periodic behavior of a feature. The vector representation is used to modify time inputs in machine learning architectures.