RAID Storage Strategy for Machine Learning Training Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current machine learning model training is hindered by inefficient storage and delivery of training data, leading to performance and accuracy issues due to inadequate storage strategies that do not account for the varying usefulness of data blocks, resulting in capacity wastage and delayed deployment.

Innovation Solution

A computer-implemented method determines a storage strategy for each data block in a training dataset based on a computed usefulness score, employing RAID strategies like striping, mirroring, parity, and double parity to optimize storage, redundancy, and performance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If a uniform storage strategy is applied to all training data blocks, then storage management is simplified, but capacity wastage occurs due to not accounting for varying usefulness of different data blocks

Engineering Contradiction:

Improvestorage management simplicityVSAvoidstorage capacity wastage

Core Design Contradiction:

Ease of manufactureVSLoss of substance

Solution Approach 1:

The patent segments the training dataset into multiple data blocks and further divides each data block into individual data items. This segmentation enables differentiated storage strategies to be applied to different data blocks based on their usefulness scores, thereby avoiding uniform storage allocation and reducing capacity wastage while maintaining manageable complexity through systematic classification.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements local quality by assigning different RAID storage levels (RAID 0, 1, 5, 6, 10, etc.) to different data blocks based on their computed usefulness scores. High-usefulness data blocks receive higher redundancy levels while low-usefulness blocks use simpler storage strategies. This localized differentiation optimizes storage capacity utilization without overwhelming complexity.

Inventive Principle:

Principle #3Local quality

2Reliability

If high redundancy storage strategies are applied to all data blocks, then data reliability is improved, but storage capacity is wasted on less important data

Engineering Contradiction:

Improvedata reliabilityVSAvoidstorage capacity utilization

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent changes the redundancy parameter (RAID level) dynamically based on the usefulness score of each data block. Data blocks are scored based on their importance to model training, and the RAID level is adjusted accordingly - high-scoring blocks receive RAID 5, 6, or 10 for high reliability, while low-scoring blocks use RAID 0 or 1. This parameter adaptation ensures optimal balance between reliability and storage capacity utilization.

Inventive Principle:

Principle #35Parameter changes

3Device complexity

If data is stored without chunking into data blocks, then storage structure is simpler, but access efficiency and training performance are reduced

Engineering Contradiction:

Improvestorage structure complexityVSAvoidtraining data access efficiency

Core Design Contradiction:

Device complexityVSProductivity

Solution Approach 1:

The patent segments the training dataset into multiple chunked data blocks that can be independently processed and stored. This chunking enables parallel access to different data blocks during training, improving data access efficiency and overall training productivity. The segmented structure also facilitates selective loading of high-usefulness data blocks, further enhancing training performance without excessive complexity.

Inventive Principle:

Principle #1Segmentation

4Productivity

If dynamic storage strategies based on usefulness scores are implemented, then storage efficiency is improved, but system complexity increases

Engineering Contradiction:

Improvestorage efficiencyVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent implements preliminary action by pre-computing usefulness scores for all data blocks before applying storage strategies. This upfront calculation of data importance metrics allows the system to establish a clear classification framework that guides subsequent storage decisions. By performing this analysis in advance, the system avoids complex real-time decision-making during training operations, thereby improving storage efficiency while controlling system complexity.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11809373B2Defining redundant array of independent disks level for machine learning training data

Publication Date: 2023.11.07 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

One or more computer processors determine a storage strategy for each chunked data block in a training dataset based on a respective computed usefulness score and a series of usefulness thresholds, wherein the storage strategy comprises RAID strategies that include striping, mirroring, parity, and double parity. The one or more computer processors distribute each data block in the training dataset according to the respective determined storage strategy.