Epoch Date Indexing and Binary Search for Feature Engineering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for calculating date differences in feature engineering require extensive computational resources and storage due to the need for large matrices and duplicate calculations, which are inefficient and time-consuming.
Innovation Solution
Utilizing data structures that convert dates to epoch format and employ one-dimensional arrays to determine date proximity, eliminating the need for matrix calculations and duplicate processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional matrix calculation methods are used to determine date differences, then complete date proximity information can be obtained, but computational resources and processing time increase significantly
Solution Approach 1:
The patent pre-calculates and stores the difference between each date of interest and target dates in a sorted list during data preparation. This preliminary action eliminates the need for repeated matrix calculations during model training, significantly improving processing speed while maintaining accuracy.
Solution Approach 2:
The patent replaces the mechanical matrix calculation system with a more efficient data structure approach using sorted lists and binary search. This substitution reduces computational complexity from O(D×H) to O(D×log H), dramatically improving productivity.
2Loss of information
If large matrices are used to store date difference values, then all date proximity information can be stored, but storage requirements increase significantly
Solution Approach 1:
The patent extracts only the necessary date difference information from the complete matrix and stores it in a compressed sorted list format. This extraction retains all required proximity information while dramatically reducing storage requirements by eliminating redundant data.
Solution Approach 2:
The patent discards the large matrix structure and recovers the essential information through a more efficient data representation. By storing only the sorted difference values and using binary search, the system recovers complete date proximity information with minimal storage allocation.
3Reliability
If duplicate calculations are performed in conventional methods, then all possible date differences can be computed, but processing time increases significantly
Solution Approach 1:
The patent performs the date difference calculations once during data preparation and stores the results in a sorted list. This preliminary action ensures reliability by computing all necessary differences upfront, while eliminating duplicate calculations during model training that would otherwise consume excessive time.
Solution Approach 2:
The patent creates a sorted list copy of the date difference values that can be efficiently queried without recalculating. This copying approach maintains calculation completeness while avoiding redundant computational operations, significantly reducing processing time.
Data Source
AI summary
Methods and systems that improve upon the speed of engineering features related to date differences. Such methods and systems comprise one or more data structures that can use epoch dates to establish an index in the data structure, and integer differences as values in the data structure. The data structures can then be used as a lookup for a date of interest, to efficiently determine the number days since a previous target date and the number of days until a next target date.