Record Shuffling via Token Sorting for ML Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Business organizations face barriers in utilizing machine learning due to the requirement of statistical and AI expertise, computational intensity, and the challenge of analyzing large datasets for accurate predictions, especially for infrequent events like financial fraud detection.
Innovation Solution
A customizable machine learning service (MLS) with programmatic interfaces that guides non-expert users through machine learning best practices, allowing default settings and enabling expert customization, along with a provider network for scalable and secure data processing, and asynchronous job scheduling to manage large datasets efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning techniques are used to analyze large datasets for accurate predictions, then prediction accuracy is improved, but computational resource requirements increase
Solution Approach 1:
The patent segments large datasets into smaller batches for processing. The machine learning model processes data in distributed batches across multiple computing nodes, reducing the computational burden on individual nodes while maintaining overall prediction accuracy through aggregated results from multiple processing passes.
2Reliability
If random shuffling of observation records is applied to ensure independence of records, then model training quality is improved, but I/O overhead increases
Solution Approach 1:
The patent applies preliminary action by pre-shuffling the entire dataset before splitting it into batches. This ensures that records are randomly ordered at the outset, maintaining independence across batches without requiring repeated shuffling operations during training. The shuffle is performed once upfront, reducing cumulative I/O overhead while preserving the statistical independence needed for reliable model training.
3Stability of the object's composition
If consistent shuffling is applied across multiple passes to ensure reproducibility, then result consistency is improved, but system complexity increases
Solution Approach 1:
The patent uses parameter changes by controlling the random seed parameter across multiple training passes. By setting and maintaining a fixed random seed value, the system ensures that the same shuffle order is applied consistently across all passes, achieving reproducible results. This simple parameter control mechanism avoids complex synchronization systems while ensuring consistency.
4Measurement precision
If expert knowledge in statistics and AI is required to develop machine learning models, then model quality is improved, but ease of use deteriorates
Solution Approach 1:
The patent implements self-service by providing automated machine learning model generation that does not require expert intervention. The system automatically selects appropriate algorithms, configures parameters, and processes data through standardized workflows, enabling business analysts and subject matter experts to develop functional models without deep statistical or AI knowledge. This democratizes access to machine learning while maintaining acceptable model quality for practical applications.
Data Source
AI summary
A determination that a machine learning data set is to be shuffled is made. Tokens corresponding to the individual observation records are generated based on respective identifiers of the records' storage objects and record key values. Respective representative values are derived from the tokens. The observation records are rearranged based on a result of sorting the representative values and provided to a shuffle result destination.