Storage Device ML Facet Filtering for Dataset Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage devices struggle to efficiently filter out noisy, redundant, and superfluous data from datasets, leading to performance bottlenecks and reduced quality of machine learning models due to the transfer of large, unfiltered datasets.
Innovation Solution
Storage devices utilize ML facets to automatically filter datasets by applying dataset preparation operations based on ML facet mappings, providing high-quality, filtered datasets to machine learning applications, and allowing user customization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large unfiltered datasets are transferred from storage to ML models, then complete data is provided for training, but retrieval performance is severely bottlenecked and training time increases
Solution Approach 1:
The system extracts and removes noisy, redundant, and superfluous data from datasets before transfer, keeping only the useful portions needed for ML training. This extraction process resolves the contradiction by providing complete relevant data while eliminating unnecessary data that causes retrieval bottlenecks.
Solution Approach 2:
The storage device performs dataset filtering and preparation operations before the ML application requests the data. By preliminarily processing the dataset to remove irrelevant portions, the system ensures that only useful data is transferred, improving retrieval performance without compromising data completeness for training.
2Quantity of substance
If large unfiltered datasets are transferred, then all available data is available for training, but network and computational resources are wasted on irrelevant data
Solution Approach 1:
The system extracts and removes noisy, redundant, and superfluous data from datasets before transfer, keeping only the useful portions needed for ML training. This extraction process resolves the contradiction by providing complete relevant data while eliminating unnecessary data that causes retrieval bottlenecks.
Solution Approach 2:
The system discards irrelevant, noisy, and redundant data portions from datasets, recovering and retaining only the valuable information needed for ML training. This discarding process improves resource efficiency by preventing waste of network and computational resources on useless data while maintaining data availability for effective training.
3Manufacturing precision
If storage devices filter datasets automatically, then dataset quality improves and training efficiency increases, but system complexity increases
Solution Approach 1:
The storage device autonomously performs dataset filtering and preparation operations using predefined ML facet mappings without requiring external intervention. This self-service capability improves dataset quality and training efficiency while managing system complexity by automating the filtering process through pre-established rules and mappings.
Solution Approach 2:
The storage device performs dataset filtering and preparation operations before the ML application requests the data. By preliminarily processing the dataset to remove irrelevant portions, the system ensures that only useful data is transferred, improving retrieval performance without compromising data completeness for training.
4Adaptability or versatility
If users can customize dataset preparation, then flexibility and adaptability improve, but ease of operation decreases
Solution Approach 1:
The storage device autonomously performs dataset filtering and preparation operations using predefined ML facet mappings without requiring external intervention. This self-service capability improves dataset quality and training efficiency while managing system complexity by automating the filtering process through pre-established rules and mappings.
Data Source
AI summary
Examples described herein relate to preparing datasets in a storage device for machine learning (ML) applications. Examples include maintaining ML facet mappings between ML facets and dataset preparation tags, deriving ML facets of a dataset stored in the storage device, and generating filtered datasets from the datasets using the ML facets and ML facet mappings. The filtered dataset is associated with improved dataset quality compared to unfiltered dataset. The storage device transmits the filtered dataset to ML applications requesting the dataset. Some examples include recommending, by the storage device, ML facets to the ML application based on performance metrics.