Key-Value Engine Sampling and Shuffle for AI Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data processing approaches for AI training involve significant computational overhead and bottlenecks due to data movement between storage systems and processing components, struggling to accommodate diverse AI workloads and specialized hardware integration.
Innovation Solution
Perform sampling and shuffle operations directly within a key-value storage engine using an enhanced iterator initialization function that accepts parameters for randomization, employing an invertible hash function and dedicated processors like ASIC or FPGA to streamline data management and access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is moved between storage systems and processing components for sampling and shuffle operations, then data preprocessing can be performed, but computational overhead and data movement bottlenecks increase
Solution Approach 1:
The patent combines the key-value storage engine with sampling and shuffle operations into a single integrated system. The storage engine performs these operations internally without requiring data to be moved to separate processing components, thereby eliminating data movement bottlenecks and reducing computational overhead.
Solution Approach 2:
The key-value storage engine performs sampling and shuffle operations on its own data without external intervention. The iterator initialization function with sampling ratio and seed parameters enables the storage engine to autonomously perform these preprocessing operations, eliminating the need for separate processing components and reducing data movement.
2Adaptability or versatility
If conventional key-value storage engines are used, then basic data storage and retrieval operations are supported, but sampling and shuffle operations require separate processing components
Solution Approach 1:
The key-value storage engine is enhanced to perform multiple functions: basic storage, retrieval, iteration, sampling, and shuffle operations. The iterator initialization function accepts additional parameters (sampling ratio, seed) that enable the same storage engine to perform diverse data preprocessing operations without requiring separate specialized components.
3Reliability
If data is processed through multiple stages with separate components, then comprehensive preprocessing is achieved, but resource utilization becomes inefficient
Solution Approach 1:
Multiple preprocessing stages (storage, iteration, sampling, shuffling) are merged into a single integrated key-value storage engine. This eliminates the need for separate processing components and intermediate data transfers, reducing computational resource utilization while maintaining complete preprocessing functionality through the enhanced iterator operations.
Data Source
AI summary
The present disclosure provides a method for performing sampling operations within a key-value storage engine for AI training workflows, comprising organizing data as key-value pairs within the key-value storage engine, where each key is stored in memory and points to a corresponding value stored in a storage unit, implementing an enhanced iterator initialization function that accepts a database name, a start key, a sampling ratio parameter that determines a proportion of data to be scanned from an entire database, and a seed parameter that serves as a randomization seed, executing a random permutation over a subset of the dataset based on the sampling ratio and seed parameters, and returning values based on the randomized permutation using iterator operations, thereby performing sampling operations directly within the key-value storage engine without requiring intermediate data transfers.