High-speed parser for scalable machine learning data collection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning approaches face challenges with large datasets that exceed the memory limits of a single server, leading to costly distributed computing solutions or time-consuming data reduction methods that compromise model performance.
Innovation Solution
A high-speed parser is deployed early in the machine learning pipeline to collect statistics and identify proxies for feature importance, reducing memory usage by selectively loading only relevant data, thereby allowing larger datasets to be processed on a single server with reduced hardware requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed computing technologies are used to process large datasets, then memory capacity is improved, but device complexity and operational overhead increase
Solution Approach 1:
The patent segments the data processing task into multiple passes over the dataset, where each pass collects specific statistics. This allows a single server to process large datasets by breaking down the computation into manageable stages rather than requiring all data to be loaded into memory simultaneously.
Solution Approach 2:
The patent performs preliminary scanning passes to collect data statistics before the actual machine learning training. This preliminary action identifies important features and data characteristics, allowing the system to optimize subsequent processing and reduce the memory requirements for the main training operation.
2Quantity of substance
If manual data removal is performed to fit datasets in memory, then hardware requirements are reduced, but processing time and model performance deteriorate
Solution Approach 1:
The system performs preliminary scanning passes to automatically identify and collect statistics about data features, eliminating the need for manual data selection. This automated preliminary analysis reduces data preparation time while maintaining model performance by objectively identifying important features.
Solution Approach 2:
The data processing system performs self-analysis through automated statistical collection and feature importance identification, eliminating the need for manual human intervention in data selection and preparation processes.
3Reliability
If entire datasets are loaded in memory for machine learning training, then model accuracy is improved, but hardware resource requirements increase
Solution Approach 1:
The patent segments the dataset processing into multiple passes, with each pass collecting specific statistical information. This allows the system to process large datasets that cannot be loaded entirely into memory, while still maintaining model accuracy through comprehensive statistical analysis of all data.
Solution Approach 2:
The system performs preliminary scanning passes to collect data statistics and identify important features before actual training. This preliminary action ensures that only the most relevant data characteristics are used in training, maintaining model accuracy while reducing memory requirements.
4Measurement precision
If multiple scanning passes are performed on raw data, then data analysis completeness is improved, but processing time increases
Solution Approach 1:
The patent segments the data analysis into multiple specialized passes, where each pass collects specific types of statistics. This segmentation allows for comprehensive data analysis while optimizing processing time by focusing each pass on specific measurement goals rather than redundant full analyses.
Solution Approach 2:
The system maintains continuous useful action by performing multiple scanning passes without reloading data, keeping the data stream active and continuously collecting different statistical measures in sequence, maximizing the utility of each data read operation.
Data Source
AI summary
A parser is deployed early in a machine learning pipeline to read raw data and collect useful statistics about the raw data's content to determine which items of raw data exhibit a proxy for feature importance for the machine learning model. The parser operates at high speeds that approach the disk's absolute throughput while utilizing a small memory footprint. Utilization of the parser enables the machine learning pipeline to receive a fraction of the total raw data that would otherwise be available. Several scans through the data are performed, by which proxies for feature importance are indicated and irrelevant features may be discarded and thereby not forwarded to the machine learning pipeline. This reduces the amount of memory and other hardware resources used at the server and also expedites the machine learning process.